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Abstract 


An  After  Action  Review  (AAR)  is  the  Army  training  system’s  doctrinal  feedback 
mechanism.  The  purpose  of  the  AAR  is  to  improve  collective  (imit)  and  individual 
performance  in  order  to  enhance  organizational  readiness.  It  is  a  learning  process.  While 
the  literature  discusses  instructional  and  training  systems,  neither  the  AAR  process  nor 
AAR  systems  have  been  examined  in  terms  of  learning  effectiveness  and  efficiency. 

In  this  thesis,  four  elements  that  combine  to  produce  an  effective  AAR  (one  in 
which  the  trainees  learn)  are  derived  from  the  literature.  A  methodology  to  measure  AAR 
effectiveness  with  respect  to  these  elements  is  applied  to  17  Combat  Training  Center 
AARs.  Results  of  this  research  suggest  that  AAR  effectiveness  can  be  improved.  An 
approach  based  upon  “guided  discovery  learning”  that  takes  advantage  of  current 
advances  in  training  simulation  technology  is  presented  for  implementation  within  the 
Army  training  system.  Research  suggests  this  approach  will  facilitate  learning  from  a 
recent  training  experience  and  enhance  the  effectiveness  of  the  AAR. 


To  those  who  train  to  learn, 
may  they  never  have  to 
give  their  lives  in  vain. 
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PREFACE 


Revolution  in  Military  Training 

Since  the  early  1970’s,  the  US  Army  has  experienced  a  revolution  in  training  for 
war  that  has  increased  its  effectiveness  many  fold.  This  increase  in  training  effectiveness 
is  revolutionary  because,  for  the  first  time  in  history,  an  army  has  the  capability  to 
accurately  assess  its  own  performance  and  then  implement  changes  to  improve.  Prior  to 
this,  accurate  training  assessment  depended  upon  the  experience  of  the  leader  in  charge. 
This  revolutionary  ability  can  be  directly  attributed  to  an  enhanced  combat  simulation 
capability,  a  standardized  and  performance-oriented  task-based  training  language,  and  a 
formalized  mechanism  for  performance  feedback.  The  most  apparent  result  of  this 
revolution  is  the  Combat  Training  Centers  (CTC)  in  Ft  Polk,  LA,  Ft  Irwin,  CA,  and 
Hohenfels,  FRG.  These  CTCs  provide  realistic  conditions  and  prompt  performance 
feedback  for  units  training  there.  The  CTCs  are  the  most  effective  training  systems  the 
US  Army  has  ever  devised  and  they  were  a  major  factor  in  the  Army’s  success  in  Panama 
and  Iraq. 

Prior  to  the  mid  1970’s,  training  was  characterized  by  three  conditions.  First, 
training  conditions  were  unrealistic  compared  to  actual  battlefield  conditions.  Second, 
unit  actions  were  ill-defined  in  terms  of  individual  and  collective  tasks  and  had 
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correspondingly  vague  performance  standards.  Third,  performance  feedback  was 
nonstandard  across  the  Army.  While  objective  performance  feedback  and  assessments 
were  provided  to  individuals  in  the  form  of  a  critique,  collective  feedback  and 
assessments  were  based  upon  subjective  and  vague  criteria  such  as  the  amount  of  noise  a 
unit  produced  during  an  assault.  (Collins,  A.  S.,  1978) 

In  the  latter  half  of  the  1970’s,  improvements  in  combat  effects  simulation,  most 
notably  the  laser  engagement  system  in  use  for  direct  fire,  provided  realistic  performance 
outcomes.  This  in  turn  allowed  the  imits’  actions  employing  those  weapons  to  be  verified 
and  validated  in  peace  time  training.  Heretofore,  this  verification  and  validation  only 
occurred  in  actual  combat.  In  peacetime,  this  evaluation  of  performance  and  its  feedback 
to  the  training  unit  was  the  sole  domain  of  combat  veterans  whose  experience  varied  to 
great  degrees.  Moreover,  the  realistic  execution  of  individual  and  collective  combat 
actions  during  peacetime  allowed  detailed  task  analyses  to  be  conducted.  This  resulted  in 
unit  actions  being  described  in  terms  of  collective  tasks  consisting  of  supporting 
collective  and  individual  tasks/drills.  This  hierarchy  of  tasks  is  recorded  in  the  Army’s 
Mission  Training  Plan  (MTP)  series  of  references  and  is  the  language  (medium  of 
communication)  used  to  assess  and  effect  performance  improvement.  Consequently,  as 
task  execution  became  more  descriptive,  performance  feedback  became  more  detailed 
and  formalized  in  delivery.  This  feedback  mechanism,  called  the  After  Action  Review 
(AAR),  signified  the  Army’s  training  process  as  a  system.  With  the  integration  of 
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combat  simulation  improvements,  the  AAR  delivery  method  evolved  from  a  subjective 
critique  to  participatory  discussion  of  performance. 

Improvements  in  computer-based  simulator  technology  will  sustain  the 
revolutionary  wave  in  two  ways.  First,  these  improvements  will  have  the  same  impact  on 
command  and  staff  operations  as  the  tactical  engagement  system  technology 
improvements  had  on  unit  combat  engagement  actions.  The  increase  in  realistic 
simulations  will  allow  command  and  staff  operations  to  be  converted  into  a  task-based 
knowledge  domain.  Most  importantly,  standards  of  performance  can  be  assigned  to  these 
operations  and  fed  back  to  the  imits. 

The  second  impact  of  improved  simulator  technology  is  its  enhancement  of  the 
feedback  and  feedback  mechanism.  Technology  improvements  allow  more  performance 
data  to  be  collected  and  presented  at  a  faster  rate  than  ever  before.  Hence,  these 
improvements  directly  affect  the  AAR’s  preparation  effectiveness  (collecting  and 
presenting  the  important  data)  and  efficiency  (sooner  rather  than  later). 

These  conditions  have  placed  the  Army  in  a  propitious  position  for  further 
improvement  and  participation,  as  an  agent  of  change,  in  the  early  stages  of  a  nationwide 
educational  revolution. 

Revolution  in  Learning 

With  guided  discovery  learning,  the  military  is  unobtrusively  on  the  crest  of  a 
revolutionary  wave  in  education.  Since  still  early  in  evolution,  it  is  probably  more 
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appropriate  to  say  that  military  research  and  development  funds  provide  the  sustaining 
energy  to  the  wave.  With  current  budget  cuts  and  military  draw-down,  the  sustenance  has 
been  reduced  to  a  number  of  iterative  efforts  producing  only  small  wave  action.  As  these 
choppy  research  efforts  converge  and  diverge  outside  the  foundations  of  our  nation’s 
educational  edifice,  the  walls  are  also  being  worn  from  within.  Most  of  this  internal  strife 
centers  on  what  is  learned.  This  strife  is  a  constant  drain  and  source  of  divergence  for  the 
revolutionizing  wave  of  how  one  learns.  As  cognitive  learning  and  meta-cognitive 
research  clarify  how  humans  learn,  the  what  to  learn  question  will  decrease  in  importance 
and  only  surface  when  a  choice  of  subjects  is  available.  The  how  will  also  define  what 
tools  are  needed  to  learn. 

Notwithstanding  the  current  situation  in  civilian  education,  the  Army  training 
system  can  improve  and  provide  a  path  to  follow.  The  AAR  is  the  Army’s  mechanism  to 
improve  readiness  and  correct  performance  deficiencies.  Indeed,  the  AAR  is  the 
mechanism  by  which  individuals  and  rmits  learn.  This  thesis  proposes  a  novel  approach 
to  increase  the  effectiveness  of  Army  AAR  by  employing  a  meta-cognitive  instructional 
strategy  and  utilizing  the  advantages  provided  by  computer-based  simulation  training 
systems. 

Organization  of  the  Thesis 

As  with  most  at  the  masters  level,  this  thesis  is  an  iterative  and  exploratory  effort. 
Chapter  I  explains  the  background  and  definition  of  the  AAR.  How  the  AAR  fits  into  the 
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Army  training  system,  its  origins,  and  its  doctrinal  definition  are  outlined.  The  sources 
drawn  upon  for  this  information  are  Army  doctrinal  references  (field  manuals  and 
training  circulars)  and  a  number  of  references  from  the  US  Army  Research  Institute 
(ARI).  Personal  communications  with  first  hand  sources  were  also  used.  Dr.  Robert 
Sulzen,  ARI  Ft  Benning,  GA  field  office,  and  Dr.  Larry  Meliza,  ARI  Orlando,  FL  field 
office,  provided  both  color  and  context  to  the  references  and  technical  reports  concerning 
the  AAR’s  origins  and  early  research. 

Chapter  II  reports  the  literature  review  to  determine  what  makes  an  AAR 
effective.  This  research  considered  the  purpose  of  the  AAR  in  light  of  AAR  research, 
performance  feedback  (behavioral  psychology),  and  cognitive  learning  theory  literature  to 
determine  the  elements  of  an  effective  AAR.  Most  of  the  AAR  research  was  conducted 
or  contracted  by  ARI.  AAR  literature  review  research  has  been  recently  contracted  by  the 
US  Army  STRICOM.  Personal  subject  matter  experience  is  also  cited  to  confirm  or 
question  the  findings  of  the  review.  The  elements  required  to  produce  an  effective  AAR 
are  stated. 

Chapter  III  defines  the  problem.  The  AAR’s  current  state  and  condition  are 
described  by  reviewing  what  the  components  of  an  AAR  are  at  the  CTCs  and  in  small 
unit  simulation  training  systems.  AAR  systems  at  the  JRTC  and  National  Training 
Center  (NTC)  are  outlined.  Additionally,  the  Close  Combat  Tactical  Trainer  (CCTT)  and 
the  Simulation  Network  (SIMNET)  AAR  systems  are  analyzed  with  respect  to  the  CTC’s 
needs  and  the  Army’s  21®*  century  plan  for  a  standard  Army  AAR  system  (STAARS). 
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The  simulation  research  improvement  AAR  systems  analyzed  are  the  Automated 
Training  Analysis  Feedback  System  (ATAFS)  and  the  Simulation  Training  Integrated 
Performance  Evaluation  System  (STRIPES)  built  by  LBM  Inc.  and  AcuSoft  Inc. 
respectively.  The  sources  cited  are  the  CTC  AAR  guidance  documents  and  the 
development  guidance/design  dociunents  for  each  of  the  simulation  systems.  Finally, 
using  the  elements  of  effectiveness  derived  in  Chapter  II,  a  methodology  was  developed 
to  assess  effectiveness  in  an  actual  AAR.  Seventeen  formal  platoon  and  company  level 
AARs,  videotaped  at  the  JRTC  and  NTC,  were  analyzed. 

Chapter  IV  presents  an  approach  to  increase  AAR  effectiveness  in  small  unit 
simulation  training  systems.  This  approach  derives  from  intelligent  tutoring  research  to 
teach  problem  solving  and  is  particularly  suited  to  support  simulation  based  training.  A 
vision  of  how  the  approach  can  be  implemented  as  a  system  is  presented  in  Appendix  E. 

Chapter  V  summarizes  the  findings  and  limitations  of  the  research  as  well  as 
identifies  areas  of  potential  future  research. 

Subject  Matter  Experience 

Throughout  this  paper,  I  cite  my  training  and  AAR  experience  to  temper  the 
research  with  reality.  This  subject  matter  experience  is  the  result  of  1 1  years  of  Army 
service.  In  that  time,  I  have  acted  in  a  number  of  company  grade  leadership  and  two 
battalion  level  staff  positions.  In  these  positions,  I  have  acted  as  both  trainer  and  trainee 
in  mechanized,  air  assault,  airborne,  and  light  infantry  units.  The  majority  of  my  AAR 


xxii 


experience  was  gained  in  the  last  five  years  of  service.  Prior  to  this,  I  was  assigned  to 
units  in  which  trainiging  was  not  systematic.  That  is,  the  unit  training  systems  did  not 
actively  employ  or  accept  the  AAR  as  the  primary  performance  feedback  mechanism.  In 
the  33  months  of  two  company  commands,  I  conducted  and  supervised  numerous  formal 
and  informal  AARs.  After  command,  I  spent  two  years  at  the  Joint  Readiness  Training 
Center  (JRTC)  at  Fort  Polk,  LA,  as  an  observer/controller  (OC).  During  this  time,  I  led 
and  facilitated  65  formal  AARs  and  numerous  informal  AARs. 
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CHAPTER  I 


ORGANIZATIONAL  SETTING 


"The  old  saying  'Live  and  Leam'  must  be  reversed  in  war,  for  there  we 
'Learn  and  Live;'  otherwise,  we  die.  It  is  witii  this  learning, 
in  order  to  live,  that  the  Army  is  so  vitally  concerned." 

—  US  War  Department  Pamphlet  No.  20-17 
July  1945 

Learning  and  the  Army  Training  System 

Doctrine 

The  purpose  of  the  US  Army  is  to  maintain  peace  by  deterring  war  or,  if  needed, 
to  “reestablish  peace  through  victory  in  combat  wherever  US  interests  are  challenged.” 
(Department  of  the  Army  [DA],  FM  25-100, 1988b,  1-1)  Training  is  the  cornerstone  that 
allows  the  US  Army  to  achieve  its  purpose  and  assigned  missions.  It  is  the  focal  point  of 
peacetime  operations  and  a  vital  component  of  wartime  operations.  Training  is  the  single 
common  thread  that  runs  through,  maintains,  and  strengthens  the  fabric  that  is  the  US 
Army. 
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The  Army’s  training  doctrine  is  presented  in  two  references  -  Field  Manual  (FM) 


9S-1  on  Training  the  Force,  and  FM  25-101.  Battle  Focused  Training.  FM  25-100 
describes  the  US  Army’s  overarching  training  doctrine  that  is  applicable  throughout  the 
organization;  individual  through  unit  level.  Its  central  theme  is  nine  principles  of  training 
that  define  the  training  management  system.  These  principles  are  the  framework  upon 
which  all  individual  and  collective  training  is  constructed.  They  are  used  as  guidelines 
for  leaders  responsible  for  planning,  preparing,  and  executing  training.  These  principles 
of  training  are  summarized  from  FM  25-100  below.  (DA,  FM  25-100, 1988b,  1-3  through 
1-5) 

Train  as  combined  arms  and  services  team  -  this  principle  acknowledges  that 
success  in  wartime  is  dependent  upon  a  number  of  combat,  combat  support,  and  combat 
service  support  units  within  the  Department  of  Defense.  The  Army  has  the  capability  to 
collectively  train  the  commanders  and  units  who  will  eventually  deploy  and  fight 
together.  The  objective  is  not  to  make  the  battlefield  the  location  where  units  meet  to 
work  together  for  the  first  time. 

Train  as  you  fight  -  simply,  this  is  the  conduct  of  training  under  realistic 
conditions.  The  purpose  here  is  to  ensure  that  a  unit  has  practiced  all  the  actions  it  will 
execute  in  combat  prior  to  arriving  on  die  battlefield.  Critical  to  this  principle  is  the 
identification  of  the  differing  conditions  that  a  unit  may  face  in  combat. 

Use  appropriate  doctrine  —  this  ensures  that  units  are  familiar  with  and  follow 
common  procedures  and  uniform  operational  methods  that  will  minimize  reaction  time  to 
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unforeseen  situations.  It  also  provides  a  common  framework  on  which  to  plan,  prepare, 
and  execute  operations.  In  short,  it  puts  all  individuals  and  units  on  the  same  sheet  of 
music. 

Use  performance-oriented  training  —  this  principle  emphasizes  a  hands-on 
approach  in  practicing  the  critical  tasks  and  missions  that  will  be  required  in  combat. 

Train  to  challenge  —  this  principle  requires  training  conditions  to  be  intellectually 
and  physically  challenging  in  order  to  excite  and  motivate  soldiers,  hone  their  skills,  and 
increase  their  capacity  for  achievement. 

Train  to  sustain  proficiency  —  this  dictates  that  individual  and  collective  training 
must  be  conducted  at  a  frequency  and  to  the  standards  necessary  to  prevent  skill  decay, 
reinforce  teamwork  mechanisms,  and  train  new  people.  The  purpose  of  this  principle  is 
to  ensure  that  rmit  performance  is  maintained  within  “band  of  excellence”  over  time. 

The  “band  of  excellence  is  an  ideal  range  of  performance  defined  by  the  unit  commander. 
Personal  experience  has  shown  that  the  “band  of  excellence”  boundaries  are  defined 
subjectively  and  in  many  ways  by  different  commanders.  Most  commanders  will  not 
explicitly  define  the  minimum  requirements  at  all. 

Train  using  multi-echelon  techniques  -  emphasizes  maximizing  the  effective  use 
of  training  resources  by  training  on  different  levels  simultaneously.  Training  events  must 
be  structured  to  provide  critical  task  practice  for  soldiers,  leaders,  and  the  sub-units 
within  a  parent  unit.  This  approach  is  the  most  efficient  for  training  individual  and 
collective  tasks  associated  with  a  specific  critical  combat  function,  task,  or  mission. 
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Train  to  maintain  -  emphasizes  training  to  achieve  technical  proficiency  on  and 
serviceability  upkeep  of  military  equipment  and  weapons  such  that  they  are  ready  for 
war. 

Make  commanders  the  primary  trainers  -  this  principle  requires  that  the  unit 
leaders  be  held  responsible  for  the  training  and  performance  of  their  soldiers  and 
subordinate  units.  The  imit  leader  must  analyze  the  wartime  mission  requirements, 
indicate  applicable  Army  performance  standards,  assess  the  current  level  of  proficiency, 
coordinate  and  provide  the  needed  training  resources,  and  develop  and  execute  tailored 
training  plans  that  produce  proficient  subordinate  individuals  and  units. 

FM  25-101,  Battle  Focused  Training,  was  written  for  the  leaders  responsible  for 
training  at  battalion  level  and  below.  These  leaders  are  both  commissioned  and 
noncommissioned  officers  at  squad,  platoon,  company,  and  battalion  levels.  FM  25-101 
builds  on  the  doctrine  in  FM  25-100  and  provides  practical  “how  to”  guidelines  for 
leaders.  (DA,  FM  25-101,  1990,  Foreword)  It  describes  how  to  focus  a  unit’s  training 
plan  on  the  specific  tasks  that  the  unit  is  most  likely  to  perform  in  combat.  This  is  the 
“battle  focus”  of  a  training  plan.  This  concept  recognizes  that  units  do  not  have  the  time 
nor  resources  to  train  for  every  task  and  situation  that  may  arise.  (DA,  FM  25-101, 1990, 
1-10) 

Army  training  tasks  are  divided  into  individual  and  collective  tasks;  each  has 
specified  quantitative  and  qualitative  performance  measures  and  execution  standards 
associated  with  it.  Individual  tasks  are  further  subdivided  into  soldier  and  leader  tasks. 
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These  individual  tasks  combine  with  drills  to  make  up  collective  tasks.  Drills  are 
collective  tasks  that  consist  of  sequenced  individual  tasks  performed  by  a  unit  in  response 
to  a  specific  battlefield  cue.  Collective  tasks  do  not  require  a  specific  sequence  but  do 
require  the  unit  leader  to  make  one  or  more  tactical  decisions.  A  collective  task  may  also 
be  reliant  on  other  collective  tasks.  The  performance  measures  and  task  standards  are 
essential  to  define  “what”  is  being  trained  to  what  proficiency  level.  Performance 
feedback  and  training  outcomes  are  described  in  these  terms. 

Once  the  critical  tasks  to  be  trained  are  identified  and  the  nine  training  principles 
are  applied  to  plan,  prepare,  and  execute  the  training,  an  After  Action  Review  (AAR)  is 
conducted.  The  AAR  is  the  “feedback  mechanism  that  leaders  use  to  keep  the  [training] 
system  dynamic  and  capable  of  continual  improvement  and  fine  tuning.”  (DA,  FM  25- 
100,  1988b,  1-9)  It  ensures  that  performance  feedback  is  provided  to  the  individual/unit 
executing  the  training.  This  is  the  key  element  that  closes  the  loop  in  the  Army  training 
system.  It  allows  units  to  improve  performance  by  identifying  mistakes/weaknesses, 
causes,  and  the  appropriate  corrective  actions. 

To  improve  performance,  the  US  Army  relies  on  each  unit  and  individual  to  learn 
by  rigorous  practice  (to  established  performance  standards)  and  reflection  on  experience. 
The  AAR  provides  a  standard  mechanism  with  which  to  learn.  This  mechanism’s 
method  is  also  standardized  to  establish  a  common  framework  for  learning.  The  specific 
method  of  learning,  advocated  by  doctrinal  training  references,  is  a  form  of  guided 
discovery  learning.  Currently,  guided  (or  tutored)  discovery  is  the  most  efficient  learning 
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method  and  has  been  shown  to  be  more  effective  than  learning  through  instructor 
critiques.  (Williams,  1996a;  Smith- Jentsch  et  al.,  1996;  Kozlowski  et  al.,  1996;  O’Malley, 
1995;  Chapman  &  Allen,  1994;  Anderson,  1993;  Lajoie  &  Derry,  1993;  Katz  &  Lesgold, 
1993;  Newman  et  al.,  1993;  Swan  &  Black,  1993;  De  Corte  et  al.,  1992;  Lesgold  et  al., 
1992;  Shute  8c  Glaser,  1990;  and  Shriver  et  al,  1975)  The  author’s  subject  matter 
experience  confirms  this  method  and  the  basic  need  for  the  AAR.  Indeed,  without 
constructive  feedback,  individuals  and  units  will  make  the  same  mistakes  no  matter  how 
many  training  principles  are  applied  or  how  battle  focused  the  training  tasks  are.  Units 
and  unit  members  require  an  AAR  to  learn  and  consequently  effect  improved 
performance. 


The  System  Implemented 

Today,  AARs  are  an  integral  part  of  training  at  all  levels.  This  is  because  the 
Combat  Training  Centers  (CTC)  have  institutionalized  the  AAR  as  part  of  the  training 
process.  The  most  realistic  combat  training  occurs  at  the  CTCs  —  the  National  Training 
Center  (NTC)  at  Fort  Irwin,  California,  the  Joint  Readiness  Training  Center  (JRTC)  at 
Fort  Polk,  Louisiana,  and  the  Combat  Maneuver  Training  Center  (CMTC)  at  Hohenfels, 
Germany.  These  training  centers  evolved  out  of  the  Tactical  Engagement  System  (TES) 
concepts  of  the  1970s.  The  CTCs  are  locations  where  maneuver  brigades  consisting  of 
two  combat  task  forces  and  their  combat  support  and  combat  service  support  units  can 
immerse  themselves  in  a  simulated  combat  environment.  Each  direct  fire  weapon  system 
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is  instrumented  to  simulate  its  signature  and  effect.  Others,  such  as  indirect  fire  weapon 
systems  require  supporting  personnel  and  systems  to  simulate.  Combat  arms  units  can 
practice  most  missions  and  tasks  that  they  may  perform  in  combat  with  all  their  weapons 
and  equipment. 

The  realistic  training  conditions  also  include  terrain  with  minimal  maneuver 
restrictions  and  a  highly  trained  opposing  force  (OPFOR).  Efforts  are  taken  to  ensure 
that  the  OPFOR  is  better  trained  than  the  unit  rotated  in  for  training.  The  OPFOR  also 
has  the  advantage  of  knowing  the  terrain,  whereas  the  terrain  is  relatively  unfamiliar  to 
the  rotational  unit.  The  premise  is  that  a  unit  will  improve  more  efficiently  if  they  train 
against  a  better  unit. 

The  missions  conducted  against  the  OPFOR  are  “ffee-play”  training  exercises. 
Units  must  analyze  their  mission  with  respect  to  the  enemy,  terrain,  firiendly  troops,  and 
time  available  and  then  devise  and  execute  a  plan.  At  the  same  time,  the  OPFOR  are 
given  a  contradicting  mission  and  must  perform  the  same  analysis  to  formulate  their  own 
plan.  Realistic  engagement  outcomes  are  decided  through  the  use  of  instrumented 
weapon  systems  on  each  side.  These  instrumented  weapon  systems  utilize  a  laser 
engagement  system,  a  laser  strike  position  referencing  system,  and  sophisticated 
probability  of  hit/kill  algorithms  to  arbitrate  the  engagement  outcomes. 

Brigades  rotate  to  the  CTCs  for  approximately  30  days  each  year.  The  units 
spend  approximately  two  weeks  in  simulated  continuous  combat  operations  against  the 
OPFOR.  During  the  training  period,  four  AARs  are  usually  conducted  by  a  permanent 


7 


cadre  of  observer/controllers  (OC).  The  OC  teams  observe  events  and  performance  in  an 
effort  to  completely  reconstruct  a  battle  for  the  AAR.  The  realistic  training,  combined 
with  the  AARs,  make  the  CTCs  the  premier  training  experience  of  combat  forces.  Never 
before  have  units  experienced  more  realistic  training  or  learned  as  much  from  training. 
Because  of  routine  exposure,  units  have  adopted  and  institutionalized  die  battle  focused 
training  system  with  special  emphasis  on  AARs. 


The  AAR 

The  AAR  is  the  process  through  which  leaders  evaluate  and  trainees  receive 
feedback  during  and  after  individual  and  collective  training.  FMs  25-100  and  25-101  are 
the  capstone  training  references  for  the  Army.  They  each  address  the  AAR  in 
complementary  fashion.  The  formal  doctrinal  definition  of  the  AAR  is: 

a  method  of  providing  feedback  to  units  by  involving  participants  in  the 
training  diagnostic  process  in  order  to  increase  and  reinforce  learning.  The 
AAR  leader  guides  participants  in  identifying  deficiencies  and  seeking 
solutions.  (DA,  FM  25-101,  1990) 

FM  25-100  explicitly  states  that  the  AAR: 

is  a  structured  review  process  that  allows  training  participants  to 
discover  for  themselves  what  happened,  why  it  happened,  and  how  it 
can  be  done  better.  The  AAR  is  a  professional  discussion  that  requires 
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the  active  participation  of  those  being  trained.  (DA,  FM  25-100,  1988b, 

5-1) 

The  training  feedback  presented  in  the  AAR  closes  the  training  system  loop  and  enables 
units  to  improve  individual  and  collective  task  performance.  These  definitions  support 
the  goal  of  the  AAR  outlined  in  Training  Circular  (TC)  25-20.  This  goal  is  to  “improve 
soldier,  leader,  and  vmit  performance.”  (DA,  TC  25-20, 1993,  1-2) 


The  Origins  of  the  AAR 

S.  L.  A.  Marshall’s  combat  action  debriefs  of  the  Second  World  War  and  Vietnam 
were  the  impetus  for  the  modem  day  AAR.  (Sulzen,  R.,  personal  communication,  29 
August  1996)  His  debriefing  technique,  the  “after-action  interview,”  sought  to  establish 
the  ground  tmth  of  a  combat  event  for  historical  record.  (Hackworth,  1967,  and  Marshall, 
1959)  The  AAR  concept  was  borne  out  of  the  combination  of  Marshall’s  example  and 
the  combat  simulation  improvements  for  live,  squad  training  developed  in  the  late  1970’s. 
Specifically,  these  improvements  were  the  development  of  effective  casualty  assessment 
techniques  that  introduced  objectivity  into  the  training  environment.  (Zeidner  &  Dracker, 
1988,  186  to  189,  and  Shriver  et  al.,  1975, 1 1  and  12)  The  training  tools  that 
accomplished  this  were  the  Squad  Combat  Operations  Exercise  (Simulation)  (SCOPES), 


also  known  as  REALTRAIN,  and  the  Tactical  Engagement  System  (TES)  that  later 
evolved  into  the  Multiple  Integrated  Laser  Engagement  System  (MILES)  in  use  today.* 
The  SCOPES  was  originally  developed  as  a  method  to  evaluate  the  individual 
performance  under  reasonably  realistic  battlefield  conditions.  (Shriver  et  al.,  1975, 3 
through  4)  These  tools  allowed  simulation  of  realistic  battlefield  cues  and  outcomes. 
Visual  and  auditory  weapon  signatures  were  replicated  with  smoke  cartridges  and  blank 
ammunition.  For  SCOPES  (and  REALTRAIN),  each  individual  wore  an  identification 
number  on  the  helmet.  Weapon  effects  were  simulated  by  the  firer  calling  out  the  enemy 
target’s  helmet  (or  vehicle)  number  to  a  controller.  This  controller  radioed  the 
engagement  to  an  opposing  force  controller  who  assessed  the  casualty/damage. 

Later,  more  accurate  effects  were  simulated  by  activating  a  MILES  laser 
transmitter  with  the  weapon  system’s  trigger  pull.  MILES  uses  an  eye  safe  laser 
transmitter  to  replicate  the  flight  and  lethality  of  a  projectile  fired  from  a  weapon  system. 
Based  on  the  location  and  coded  lethality  of  the  laser  impact,  sensors  mounted  on  soldiers 
and  vehicles  receive  the  laser  strike  and  determine  if  the  target  is  killed,  near  missed,  or 
unaffected.  Hence,  units  could  improve  battlefield  performance  more  efficiently  by 
prescribing  performance  standards  that  were  tied  to  observable  outcomes.  (Meliza,  L., 
personal  communication,  28  August  1996,  and  Zeidner  &  Drucker,  1988, 187) 

Prior  to  SCOPES  and  the  TES,  the  training  of  combat  vmits  centered  on  the 
individual  and  crew  in  a  school  environment.^  Most  training  was  conducted  in  a 

'  The  REALTRAIN  acronym  stood  for  “real  training.” 
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“classroom-lecture”  format  both  in  a  building  and  outside  in  bleachers  or  on  the  groimd. 
(Zeidner  &  Drucker,  1988, 146,  Collins,  A.  S.,  1982,  Chapters  13  &  14,  and  Shriver  et 
al.,  1975, 2)  Collective,  force  on  force  training  outcomes  were  governed  by  umpire 
controllers.  They  decided  the  success  of  an  assault  on  a  defended  position  by 
subjectively  evaluating  the  numerical  superiority,  swiftness,  and  noise  level  (equated  to 
ferocity)  of  the  attacking  unit.  Casualties  were  rarely  simulated  and  units  did  not  practice 
consolidation  and  reorganization  on  the  objective.  Instead,  both  attackers  and  defenders 
would  stand  in  limbo  and  wait  for  the  umpire’s  decision.  (Sulzen,  R.,  personal 
communication,  29  August  1996) 

Training  feedback  was  equally  nebulous  with  subjective  decisions  governing 
exercise  outcomes.  After  the  training,  the  senior  evaluator  would  critique  the  unit.  He 
would  present  his  feelings  on  performance  indicating  that  the  unit  did  well  or  poorly 
based  on  his  interpretation  of  the  tactical  events.  Little  objective  performance  data  was 
identified  or  discussed.  This  lecture-formatted  critique  and  the  umpire’s  opinion  did  not 
effectively  convince  units  that  they  had  problems  that  required  corrective  training.  (Scott, 
1983, 1  through  2  and  Word,  1987, 34) 

Such  a  critique  is  predominantly  a  one-way  transmission  of  information  that  does 
not  facilitate  two-way  communication.  The  implied  premise  of  a  critique  is  that  the 
lecturer  possesses  all  the  facts  necessary  for  learning.  Realistically,  it  is  impossible  for 
any  participant  or  observer  to  collect  enough  information  to  enable  100%  reconstruction 


^  “Crew”  refers  to  a  vehicle  crew  that  consists  of  two  to  five  soldiers  depending  on  the  type  of  vehicle. 
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of  an  event  in  free-play  (un-scripted)  training  exercises.  Thus,  without  interviewing  each 
of  the  participants,  the  critique  becomes  grounded  in  partial  facts  and  assumptions  of 
what  happened  and  why.  Recommendations  of  corrective  actions  will  themselves  be 
partial.  Additionally,  trainees  have  difficulty  separating  criticism  from  personal  character 
attacks.  (Bosley  et  al.,  1979,  Preface) 

Given  such  a  setting,  it  is  not  surprising  that  research  psychologists  of  the  Army 
Research  Institute  (ARI)  observed  excited  reactions  in  soldiers  during  the  initial  training 
tests  with  SCOPES.  For  the  first  time,  individual  soldiers  had  control  of  the  weapon 
system’s  effect.  It  allowed  him  to  shoot  at  an  enemy  target,  score  a  hit,  and  remove  the 
opposition  from  the  exercise.  After  the  exercise,  the  friendly  force  and  OPFOR  would 
engage  in  animated  discussions  about  the  engagement.  The  seminal  elements  of  the  AAR 
evolved  out  of  the  research  psychologist’s  efforts  to  capture  this  discussion  between  the 
two  groups.  It  was  quickly  noticed  that  both  the  learning  rate  and  level  of  retention  were 
increased  by  a  controlled  discussion  between  the  attacking  and  defending  forces.  (Sulaen, 
R.,  personal  communication,  29  August  1996) 

The  introduction  of  reasonably  accurate  weapon  effects  simulations,  the 
development  of  a  responsive  casualty  assessment  technique,  and  the  research  efforts  of 
ARI  allowed  trainers  to  replace  the  critique  with  a  more  effective  teaching  technique.  To 
differentiate  it  from  the  old  lecture-format  critique,  the  new  training  feedback  method  was 
labeled  the  AAR.  The  AAR  was  developed  to  increase  teaching  and  learning 
effectiveness  through  soldier  participation.  (Scott,  1983, 2) 
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The  origins  of  the  AAR  are  recorded  in  the  Army  manuals  and  research  papers 
describing  SCOPES  and  TES.  The  AAR  was  treated  as  a  part  of  the  new  combat 
simulation  training  techniques  but  never  as  an  entity  to  be  studied  independently.  The 
research  on  SCOPES  and  TES  was  conducted  by  ARI  and/or  independent  contractors 
working  with  ARI.  In  1974,  the  US  Army  Infantry  School  (USAIS)  published  ST  7-2- 
172.  SCOPES.  Squad  Combat  Operations  Exercise  (Simulation^  A  System  for  Realistic 
Squad  TaftiV-al  Training.  It  was  one  of  the  first  official  documents  to  address  the  AAR. 

It  briefly  describes  how  an  AAR  should  be  conducted,  emphasizing  involvement  of  all 
participants  to  establish  the  truth  of  “what”  happened  in  an  engagement.  It  also  discusses 
the  AAR  leader’s  role. 

USAIS  ST  7-2-172  specifies  the  leader’s  role  as  a  discussion  facilitator  who 
guides  the  trainee  group  through  a  review  of  the  sequence  of  actions.  Each  action  is 
described  by  the  participants  in  their  own  words  to  establish  “what”  happened  for  the 
group.  The  trainees  leam  by  realizing  their  mistakes  with  respect  to  the  situation  or 
training  conditions.  (USAIS,  ST  7-2-172, 1974, 14  and  15)  Training  conditions  are 
defined  in  doctrinal  terms  of  mission,  enemy,  terrain,  troops,  and  time.  Specifically, 
these  translate  to  the  task  the  individual  or  unit  was  trying  to  achieve,  the  enemy  actions 
that  affected  performance  of  that  task,  the  terrain  considerations  —  advantages  and 
disadvantages,  other  friendly  actions  that  affected  performance,  and  the  time  available  to 
complete  the  task. 
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ARI  technical  report  S-4,  REALTRAIN:  A  New  Method  for  Tactical  Training  of 


Small  Units,  is  the  first  research  report  on  TES  during  field  trials  at  Wildflecken, 
Germany  in  1975.  (Shriver  et  al,  1975)  It  restates  the  method  described  in  ST  7-2-172 
and  highlights  that  the  AAR  is  not  a  critique.  It  also  emphasizes  full  participation  in  the 
discussion  and  the  value  of  having  weaknesses  pointed  out  by  the  opposing  soldiers 
rather  than  an  observer  or  instructor.  The  results  of  the  tests  verified  that  the  AAR  was  a 
learning  medium  for  the  soldiers  that  reinforced  appropriate  actions  and  inhibited 
inappropriate  actions.  (Shriver  et  al.,  1975, 23  through  25) 

The  tests  and  AARs  also  highlighted  the  need  for  research  on  the  utility  of 
detailed  performance  measures  for  tasks  within  a  tactical  framework.  (Shriver  et  al., 
1975, 23  through  25)  This  was  the  first  expression  of  the  need  for  training  situations 
defined  by  conditions  and  standards  in  an  official  report.  This  is  important  because  the 
conditions  of  training  (mission,  enemy,  terrain,  troops,  and  time)  and  the  explicit 
performance  measures  add  detail  and  objectivity  to  training  results  or  battle  damage 
assessment  (BDA).  The  BDA  results  serve  as  points  of  departure  for  discussion  of  cause 
and  effect  relationships,  weaknesses,  and  solutions. 

ARI  Research  Report  1219  (1979),  Development  of  Small  Combat  Arms  Unit 
Leader  Tactical  Training  Techniques  and  a  Model  Training  System,  documents  leader 
training  techniques  tested  for  effectiveness  in  TES  exercises.  This  report  is  relevant  to 
AARs  because  it  also  documents  the  inability  of  unit  leaders  to  depart  from  the  critique 
methodology  dvuing  training  feedback  sessions.  “Controllers  [AAR  leaders]  did  not 
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appear  to  be  able  to  reconstruct  the  battles  for  the  players  so  that  critical  mistakes  could 
be  identified  and  positive  actions  reinforced.”  (Shriver  et  al.,  1979, 23  and  24)  The  report 
goes  on  to  reiterate  the  guidance  foxmd  in  ST  7-2-172  and  Shriver  et  al.  (1975). 

Thomas  D.  Scott,  ARI,  wrote  the  Tactical  Engagement  Simulation  After  Action 
Review  Guidebook  in  1983  and  incorporated  the  findings  and  research  results  of  the 
1970s  in  this  “how  to”  manual  for  AARs.  Scott  outlined  procedures  for  preparing  and 
conducting  squad,  platoon,  and  company  AARs  and  contrasted  them  with  those  used  in 
the  traditional  critique.  He  emphasized  that  the  AAR  increases  soldier  participation  and 
consequently  increases:  the  scope  of  the  AAR,  the  interpretation  accuracy  of  cause  and 
effect  relationships,  and  the  trainees’  acceptance  of  performance  feedback.  Additionally, 
he  discusses  questioning  techniques  to  diagnose  training  deficiencies.  (Scott,  1983)  Scott 
also  reproduced  the  guidebook  as  a  chapter  and  appendix  to  ARI  Research  Product  84-14, 
How  to  Evaluate  Unit  Performance.  (1984). 

Scott  argued  that  participation  increased  learning  effectiveness  from  training 
feedback  for  three  reasons.  First,  active  participation,  as  opposed  to  passive  observation, 
greatly  increases  the  amoimt  of  information  the  trainee  learns  and  retains  as  shown  by 
educational  psychology  research.  Second,  trainee  participation  produces  group 
discussion  in  which  several  points  of  view  are  presented.  This  allows  the  trainees  and 
AAR  leader  to  gain  a  greater  imderstanding  of  the  problems  and  their  possible  solutions. 
Third,  participation  increases  trainee  motivation  by  providing  a  sense  of  ownership  in 
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developing  the  solutions  to  identified  problems.  This  involvement  also  reduces  an 
individual’s  resistance  to  acknowledging  his  own  mistakes.  (Scott,  1983, 2  through  3) 


The  Participatory  AAR 

The  participatory  AAR  provides  a  broader  scope  and  deeper  understanding  of 
cause  and  effect  relationships  for  identified  weaknesses  and  strengths.  Each  participant 
becomes  a  source  of  information  as  well  as  a  problem  solver.  The  critique  format  limits 
file  discussion  points,  if  there  is  any  discussion,  to  the  type  and  amount  of  information 
gathered  by  a  few  observers.  The  goal  is  to  combine  both  the  observations  and  expertise 
of  the  AAR  leader  with  the  insights  of  the  trainees  to  fiilly  define  the  actions  that 
influence  or  determine  performance.  This  combination  increases  the  accuracy  of  the 
interpretation  of  the  training  event.  Furthermore,  individuals  put  the  lessons  learned  into 
their  own  words  during  the  discussion  which,  combined  with  hearing  others  talk  about 
the  lessons,  allows  each  trainee  to  learn  and  remember  more.  (Scott,  1983, 3  through  4, 
and  Bosley  et  al.,  1979, 4  through  6) 

By  prompting  and  guiding  discussion  so  that  participants  identify  and  solve  the 
group’s  problems,  the  AAR  leader  avoids  the  negativism  associated  with  his  making 
declarative  statements.  The  AAR  leader  asks  leading  questions  of  key  players  of  an  event 
to  prompt  discussion,  bring  out  important  information,  and  make  a  learning  point. 
Prompting  discussion  to  examine  the  problem  ensures  participation  and,  consequently, 
avoids  the  situation  of  requiring  the  AAR  leader  to  directly  criticize  the  group.  This 
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criticism  is  what  normally  leads  to  resentment  and  resistance  of  the  AAR  leader  to  the 
point  where  feedback  does  not  occur.  “Because  the  information  comes  from  within  the 
group,  hostility  and  defensiveness  usually  directed  towards  the  critique  leader  are 
minimized.”  Consequently,  the  AAR  leader  can  focus  the  AAR’s  theme  on  “how  can  we 
do  it  better?”  rather  than  “what you  did  wrong”  as  is  done  in  a  critique.  (Scott,  1983,  5 
through  6) 

Participation  is  a  critical  element  that  has  been  emphasized  throughout  the 
evolution  of  the  AAR.  FM  25-101  states  that  the  AAR  is  a  “structured  review  process 
that  allows  training  participants  to  discover  [emphasis  added]  for  themselves  what 
happened,  why  it  happened,  and  how  it  can  be  done  better.”  (5-6)  The  manual  also 
outlines  an  AAR  format  consisting  of  four  parts:  1)  review  what  was  supposed  to  happen 
according  to  the  training  plan  -  training  purpose  and  objectives,  2)  establish  what 
happened,  3)  determine  what  was  right  or  wrong  with  what  happened,  and  4)  determine 
how  the  task  should  be  done  differently  the  next  time.  (DA,  FM  25-101,  1990,  5-6)  This 
format  is  “what”  must  be  discussed  in  an  AAR.  It  is  a  list  of  discussion  points  an  AAR 
leader  must  facilitate  in  order  to  meet  the  minimum  Army  standard  for  an  AAR.  Except 
for  the  recommendation  of  using  leading  questions,  the  manual  does  not  specify  the 
“how.”  It  leaves  this  up  to  the  AAR  leader  so  that  the  presentation  and  methods  can  be 
adapted  to  the  trainee  audience. 

Training  Circular  (TC)  25-20,  A  Leader’s  Guide  to  After- Action  Reviews,  is  the 
official  supplement  to  FM  25-101  on  AARs.  TC  25-20  expands  the  guidance  of  FM  25- 
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101  outlining  how  to  plan,  prepare,  and  conduct  an  AAR.  It  gives  the  definition  of  an 
AAR  as: 

a  professional  discussion  of  an  event,  focused  on  performance  standards, 
that  enables  soldiers  to  discover  for  themselves  what  happened,  why  it 
happened,  and  how  to  sustain  strengths  and  improve  on  weaknesses. .  .It 
provides — 

•  Candid  insights  into  specific  soldier,  leader,  and  unit  strengths 
and  weaknesses  from  various  perspectives. 

•  Feedback  and  insight  critical  to  battle-focused  training. 

•  Details  often  lacking  in  evaluation  reports  alone. 

Feedback  compares  the  actual  output  of  a  process  with  the  intended 
outcome.  By  focusing  on  die  task’s  standards  and  by  describing  specific 
observations,  leaders  and  soldiers  identify  strengths  and  weaknesses  and 
together  decide  how  to  improve  their  performances.  This  shared  learning 
improves  task  proficiency  and  promotes  imit  bonding  and  esprit.  (DA,  TC 
25-20,  1993,  1-1) 

It  also  explicitly  states  that  “the  AAR  is  a  problem-solving  process.  The  purpose  of 
discussion  is  for  participants  to  discover  strengths  and  weaknesses,  propose  solutions,  and 
adopt  a  course  of  action  to  correct  problems.”  (DA,  TC  25-20, 1993, 4-4)  This  is  the 
heart  of  the  AAR.  When  trainees  participate  in  a  discussion  to  achieve  this,  learning 
occurs.  When  one  of  the  elements  of  “what,”  “why,”  or  “how”  are  left  out  or  trainees  do 
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not  participate  in  the  discussion,  something  less  than  a  complete  understanding  of  the 
problem  and  its  relative  solutions  is  realized. 

Participation  is  a  major  theme  carried  throughout  the  AAR  doctrine.  (DA,  FM  25- 
100, 1988b,  FM  25-101, 1990,  and  TC  25-20,  1993)  This  is  highlighted  in  each  chapter 
of  TC  25-20  as  methods,  techniques,  and  training  aids  are  described  in  terms  of 
prompting  trainee  participation.  The  training  circular  maintains  that  the  AAR  leader 
should  endeavor  to  create  an  atmosphere  in  which  participation  is  encouraged.  The 
leader  does  this  by  entering  into  the  discussion  only  when  necessary,  constantly 
reinforcing  the  fact  that  it  is  permissible  to  disagree,  maintaining  a  focus  on  learning,  and 
encouraging  trainees  to  give  honest  opinions.  (DA,  TC  25-20, 1993, 4-1  through  4-2) 

TC  25-20  lists  key  points  that  guide  leaders  in  the  conduct  of  all  AARs.  These 
key  points  expand  the  description  given  by  FM  25-101  of  “what”  an  AAR  should 
accomplish  and  how  it  should  support  the  Army  training  system  as  a  whole.  The  key 
points  are  summarized  as  follows:  (DA,  TC  25-20,  1993,  1-3) 

•  AARs  should  be  conducted  during  or  immediately  after  each  critical  training 
event.  Critical  events  are  those  events  that  are  key  to  accomplishing  each  task  to  the 
specified  performance  standards.  (DA,  TC  25-20, 1993, 3-1  and  3-2)  These  critical 
events  are  defined  by  the  training  plan  that  consists  of  the  purpose  and  objectives  of  the 
training.  The  concept  behind  battle  focused  training  is  that  a  performance  weakness  is 
identified  and  the  corresponding  training  tasks  are  trained  to  correct  that  weakness. 
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Hence,  the  critical  events  are  defined  by  actions  that  the  unit  must  execute  to  show 
proficiency  on  the  tasks  associated  with  the  performance  weakness. 

•  AARs  should  focus  on  the  intended  training  objectives.  The  AAR  leader  must 
not  allow  issues  that  are  unrelated  to  the  purpose  of  the  training  to  be  discussed.  (DA,  TC 
25-20, 1993, 4-1  and  4-6)  The  training  objectives  are  explicitly  stated  in  the  training 
plan.  The  trainer,  the  leader  responsible  for  training,  develops  the  training  objectives 
Ifrom  the  rniif  s  training  needs.  Training  needs  are  derived  from  previous  training  and 
other  unit  assessment  tools.  They  center  on  performance  weaknesses  that  must  be 
corrected  and  strengths  that  must  be  sustained. 

•  AARs  should  focus  on  individual  and  collective  performance.  The  AAR 
leader  must  relate  performance  to  the  accomplishment  of  the  umt’s  training  objectives. 
(DA,  TC  25-20, 1993, 4-6)  Trainee  discussion  should  be  of  the  actions  that  were 
identified  as  critical  by  the  purpose  of  training.  Simply,  the  AAR  must  focus  on 
performance  because  its  overarching  goal  is  to  improve  performance. 

•  AARs  relate  to  specific  performance  standards.  When  the  trainer  develops  the 
training  plan  and  specifies  the  training  objectives,  he  also  specifies  the  standards  of 
performance  that  the  trainees  must  achieve.  This  requirement  keeps  the  discussion 
focused  on  improving  the  performance  that  the  trainer  designed  the  training  exercise  for. 
(DA,  TC  25-20, 1993, 2-4  and  3-1)  The  primary  source  for  the  collective  task  training 
standards  is  the  Army  Training  and  Evaluation  Program  Mission  Training  Plan  (AMTP 
or  MTP)  series  of  manuals.  The  sources  for  individual  task  training  standards  are  the 
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Soldier’s  Manuals  of  Common  Tasks  and  MOS  (Militaiy  Operational  Specialty)  specific 
Skill  Manuals. 

•  AARs  involve  all  participants  in  the  discussion.  When  all  participants  are 
actively  engaged  to  discover  what  happened  and  why,  they  learn  and  remember  more 
than  they  would  have  if  the  same  information  had  been  presented  in  a  lecture  format. 
(DA,  TC  25-20,  1993, 1-2)  Furthermore,  a  greater  variety  of  viewpoints,  ideas,  and 
observations  are  input  into  the  problem  solving  process.  This  will  ensure  that  problem 
areas  are  more  thoroughly  defined  and  solutions  completely  articulated. 

•  AAR  leaders  use  open-ended  and  leading  questions  to,  respectively,  prompt 
and  guide  discussion.  This  is  the  technique  recommended  to  get  the  soldiers  to  discuss 
the  desired  topic.  An  open-ended  question  is  defined  as  one  to  which  there  is  no  specific 
answer  but  allows  the  respondent  to  answer  with  what  is  significant  to  him.  (DA,  TC  25- 
20, 1993, 4-3)  This  motivates  the  trainee  to  participate  in  the  discussion  because  he  is 
allowed  to  speak  about  issues  that  he  perceives  are  important.  The  trainee  is  also  much 
less  likely  to  have  a  defensive  attitude.  (DA,  TC  25-20,  1993, 4-3)  Leading  questions  are 
those  asked  about  specific  issues  in  order  to  focus  the  discussion  on  specific  events  and 
performance.  Although  TC  25-20  does  not  explicitly  state  the  relationship,  common 
sense  dictates  that  leading  questions  should  be  used  to  orient  the  trainee  group  on  the 
topic  and  open-ended  questions  should  then  be  used  to  discover  what  happened  and  why. 
One  aim  would  be  to  get  the  participants,  not  the  AAR  leader,  to  identify  the  collective 
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perfonnance  weakness  and  explain  the  cause  and  effect  relationships  in  their  own 
vernacular. 

•  AARs  determine  strengths  and  weaknesses.  Throughout  the  discussion  the 
AAR  leader  should  note  and  svunmarize  learning  points  and  performance  weaknesses  and 
strengths.  (DA,  TC  25-20,  1993, 4-7  and  4-8)  The  intent  is  to  translate  training 
performance  into  individual,  both  soldier  and  leader,  and  collective  tasks  with  known  or 
specified  standards.  This,  in  turn,  will  define  future  training  objectives  and  the  first 
portion  of  the  explanation  of  “how  we  can  do  this  better.” 

•  AARs  link  performance  to  subsequent  training.  AARs  afford  unit  leaders  the 
opportunity  to  assess  unit  performance  and  immediately  retrain  deficient  tasks.  In  this 
manner,  fiiey  provide  a  direct  link  between  task  performance  and  execution  to  standard. 
(DA,  TC  25-20,  1993,  5-1)  The  second  portion  of  “how  we  can  do  this  better”  is  defined 
in  the  link.  The  AAR  leader  must  ensure  that  specific  training  conditions  for  the 
identified  deficient  tasks  are  articulated.  The  performance  of  the  deficient  task  under  the 
specified  conditions  should  correct  the  weakness.  In  essence,  the  participants  of  the  AAR 
are  designing  their  next  training  exercise. 

Outlined  below  is  a  recommended  AAR  format  proposed  in  TC  25-20.  (DA,  TC 
25-20,  1993, 1-3  and  Chapter  4) 

1 .  Introduction  and  rules.  The  AAR  leader  describes  the  purpose  and  sequence 
of  the  AAR.  He  also  emphasizes  that  “the  AAR  is  not  a  critique”  and  that  all  trainees 
should  participate  in  the  discussion  to  improve  performance. 
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2.  Review  training  objectives  and  intent.  The  training  objectives,  friendly  and 
OPFOR  commander’s  mission  and  intent,  and  relevant  doctrine,  tactics,  techniques,  and 
procedures  are  reviewed. 

3.  Summary  of  recent  events.  Using  leading  and  open-ended  questions,  the  AAR 
leader  guides  a  discussion  of  the  logical  sequence  of  events. 

4.  Discussion  of  key  issues.  The  AAR  leader  uses  one  of  three  discussion 
techniques  to  help  participants  discover  strengths  and  weaknesses,  develop  solutions,  and 
designate  corrective  action.  The  three  discussion  techniques  structures  are:  a 
chronological  order  of  events,  the  seven  Battlefield  Operating  Systems  (BOS),  and  key 
events/themes/issues. 

5.  Discussion  of  optional  issues.  The  AAR  leader  has  the  option  to  lead  the 
discussion  to  the  topics  of  soldier/leader  skills,  tasks  to  sustain/improve,  statistics,  or 
other  topics. 

6.  Discussion  of  force  protection  issues.  This  is  a  mandatory  topic.  It  covers  all 
aspects  of  soldier  safety  in  the  field  and  in  garrison. 

7.  Closing  comments  and  summary.  The  AAR  leader  reviews  and  summarizes 
learning  points  from  the  discussion  and  future  training  tasks  and  conditions. 

TC  25-20  differentiates  between  two  types  of  AARs  -  formal  and  informal.  Both 
types  seek  to  cover  the  same  key  points  and  utilize  the  same  format.  The  differences  are 
in  who  leads  the  AAR,  the  planning  and  preparation  time,  the  type  of  training  aids,  and 
the  location  of  the  AAR.  Formal  AARs  usually  have  observer/controllers  who  are 
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external  to  the  unit  responsible  for  leading  and  facilitating  the  AAR.  Formal  AARs  also 
are  scheduled,  take  more  time  to  plan  and  prepare,  use  complex  training  aids,  and  are 
conducted  where  best  supported.  Informal  AARs  are  normally  conducted  by  the  internal 
chain  of  command  of  the  training  unit.  They  are  conducted  as  needed  but  can  also  be 
scheduled.  Informal  AARs  take  less  time  to  prepare,  use  simple  training  aids,  and  are 
held  at  the  training  site.  (DA,  TC  25-20,  1993,  1-2  through  1-5) 

While  a  formal  AAR  may  appear  more  organized  and  well  produced  than  an 
informal  AAR,  informal  AARs  have  the  advantage  of  being  conducted  immediately  after 
training  or  a  critical  event  during  training.  This  allows  performance  feedback  to  be 
provided  while  the  training  is  still  fresh  in  the  soldier’s  minds.  Doctrine  posits  that 
soldiers  are  more  motivated  to  participate  in  the  discussion  and  will  remember  the 
learning  points  for  a  longer  period  of  time.  (DA,  TC  25-20, 1993,  1-5)  This  also  makes 
it  easier  for  the  leader  to  reinforce  learning  points  with  follow-on  discussion  and  training. 

The  training  aid  complexity  referenced  in  the  difference  between  formal  and 
informal  AARs  refers  to  the  amount  of  materials  and  support  required  to  prepare  the 
training  aid.  The  training  circular  lists  the  complex  training  aids  for  formal  AARs  as: 
terrain  model,  enlarged  map,  models,  dry-erase  marker  board,  photographs,  video  camera 
and  monitor.  The  circular  lists  the  simple  training  aids  required  for  informal  AARs  as: 
unit  markers,  pointer,  unit  maneuver  graphics,  communications  recordings,  rocks  and 
twigs,  and  colored  chalk.  (DA,  TC  25-20,  1993,  2-6  through  2-7)  Obviously,  the  training 
aids  for  formal  AARs  require  transportation  support,  electricity,  and  manpower  to  set  up. 
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The  informal  AAR  aids  can  be  carried  and  set  up  by  one  person  and  are  common  to  all 
live  training  exercises. 


Summary 

The  Army’s  training  system  is  the  tool  it  uses  to  achieve  the  state  of  readiness 
warranted  by  the  national  objectives.  This  system  is  dynamic  and  must  adapt  to  new  or 
redefined  threats  to  national  interests  and/or  domestic  need.  This  dynamic  environment, 
combined  with  a  constant  influx  of  new  (imtrained)  personnel,  requires  a  training  system 
that  can  quickly  train  units  to  the  requisite  performance  level.  The  AAR  is  the  Army’s 
doctrinal  method  of  providing  training  feedback  to  improve  individual  and  collective 
performance  to  that  specified  level.  Stated  concisely,  the  purpose  of  the  AAR  is  to 
improve  performance.  The  AAR’s  objectives  are  to: 

•  Define  the  problems  that  will  improve  performance; 

•  Articulate  the  solution(s),  in  relation  to  the  applicable  battlefield  situation,  to 

increase  learning;  and 

•  Translate  the  solutions  into  future  training  objectives  to  reinforce  learning.^ 

The  goals  and  policies  of  the  AAR  (stated  above)  are  aimed  at  creating  an 

effective  learning  environment  in  order  to  improve  performance  efficiently.  This 


^  It  is  important  not  to  confuse  the  piupose  and  objectives  of  the  AAR  with  those  of  training.  The  AAR’s 
piupose  and  objectives  never  change.  The  training  purpose  and  objective(s)  are  mission,  unit,  and  situation 
specific.  In  other  words,  training  piuposes  and  objectives  are  designed  to  meet  a  specific  training  need. 

The  AAR’s  purpose  and  objectives  are  a  component  of  the  process  used  to  meet  that  need. 
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research  is  intended  to  1)  explore  whether  or  not  the  AAR  is  implemented  I  AW  doctrine 
and  2)  propose  a  novel  approach  for  improving  the  AAR’s  effectiveness. 
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CHAPTER  II 


AAR  EFFECTIVENESS 


"Our  business,  like  any  other,  is  to  be  learned  by  constant  practice 

and  experience;" 

—  Sir  John  Moore 

What  is  an  Effective  AAR? 

Previous  Work 

A  review  of  the  AAR  literature  revealed  four  sources  that  discuss  the 
effectiveness  of  AARs.  The  first  source,  John  Bosley  et  al.  (1979),  Improved  Tactical 
Engagement  System  Training  Techniques:  Two  Training  Programs  for  the  Conduct  of 
After  Action  Reviews  (ARI  Research  Product),  deals  with  effectiveness  in  an  indirect 
manner.  The  other  three,  Larry  Word  (1987),  Observations  fi-om  Three  Years  at  the 
National  Training  Center  (ARI  Research  Product),  Cal  Downs  et  al.  (1987),  Analysis  of 
Feedback  in  After  Action  Reviews  (ARI  Technical  Report),  and  SHERIKON  Inc.  (1996), 
Analysis  of  AVCATT  After  Action  Review  Requirements  (contracted  research  report), 
address  effectiveness  directly. 
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Bosley  and  his  colleagues’  efforts  are  the  culmination  of  ARI  and  contract 
research  on  AARs  in  support  of  TES  training  exercises.  (Bosley  et  al.,  1979)  They  are 
presented  as  a  training  program  for  small  unit  leaders  conducting  AARs.  Except  for  one 
shortfall,  the  document  outlines  many  aspects  that  make  AARs  effective.  One  of  the 
significant  themes  is  trainee  participation.  Bosley  et  al.  state  that  an  effective  AAR  is  one 
in  which  the  “troops  themselves  will  have  reconstructed  the  exercise  (verbally)  the  way 
they  experienced  it,  while  the  AAR  leader  has  helped  them  connect  different  parts 
together  into  a  pattern  of  cause  and  effect  at  the  unit  level.”  (Bosley  et  al.,  1979, 6) 

Bosley  et  al.  conclude  that  the  goal  of  the  AAR  summary  is  to  show  how  prior  events  and 
actions  led  to  later  successes  or  failures  of  the  group  effort.  (Bosley  et  al.,  1979, 1 1) 

The  shortfall  of  the  research  is  its  recommendation  on  how  to  conclude  the  AAR. 
In  1979,  the  Army’s  training  management  system  (Chapter  I)  had  not  been  standardized 
across  the  organization.  Detailed  task  analyses  had  not  been  published  for  collective 
training  tasks.  Thus,  Bosley  et  al.’s  (1979)  AAR  outline  does  not  end  with  the 
development  of  a  corrective  training  plan  consisting  of  tasks  that  specifically  address 
performance  weaknesses.  This  does  not  reflect  a  lack  of  thoroughness  in  Bosley  et  al.’s 
research,  this  is  a  shortfall  in  hindsight  only. 

The  second  source  is  Observations  from  Three  Years  at  the  National  Training 
Center  by  COL  (Ret)  Larry  Word.  Word’s  research  product  is  actually  the  result  of  more 
than  a  three  year  tour  at  the  NTC.  Previously,  he  had  been  a  research  coordinator  with 
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ARI  and  had  witnessed  the  AAR’s  origins  in  the  1970s.  (Shriver  et  al.,  1975)  At  the 
NTC,  he  was  the  ehief  OC  for  mechanized  infantry  battalions  and  a  key  player  in  the 
development  of  the  AAR  system.  (Word,  1987)  As  of  1992,  his  AAR  guidance.  Word’s 
Words,  was  still  used  to  train  OCs  at  the  NTC.  (NTC/BDM,  1992)  Word  was  later 
assigned  as  the  director  of  the  JRTC  and  helped  establish  the  CMTC.  (Word,  1987) 

Word  emphasized  unit  participation  in  the  AAR  and  accuracy  of  information  as 
the  key  elements  in  an  effective  AAR.  His  conclusions  do  not  contradict  those  of  Bosley 
et  al.  However,  he  explicitly  identifies  the  OPFOR  participating  in  the  AAR  as  critical  to 
effective  learning.  The  reason  for  this  is  that  only  half  the  exercise  can  be  accurately 
reconstructed  from  half  the  participants.  The  OPFOR  provides  the  necessary  information 
that  allows  full  reconstruction.  (Word,  1987, 32  through  35)  On  the  other  hand,  Bosley 
assumes  the  OPFOR’s  presence  and  does  not  highlight  their  need  nor  importance. 

Word’s  emphasis  makes  sense  when  one  realizes  that  he  witnessed  some  of  the  first 
AARs  conducted  in  the  US  Amy. 

Word  also  stressed  that:  all  members  of  a  platoon  should  attend  and  participate  in 
the  AAR  at  that  level;  the  AAR  leader  must  adhere  to  the  format  to  keep  the  discussion 
on  track  and  limit  the  length  of  the  AAR  to  a  reasonable  amormt  of  time;  a  representation 
of  the  terrain  is  needed  to  keep  terrain  considerations  at  the  forefront  of  discussion;  any 
AAR  aids  should  clarify  problem  situations  and  illustrate  points  rapidly  and  effectively; 
and  the  AAR  should  conclude  with  a  summary  of  corrective  training  tasks  to  improve 
performance.  (Word,  1987, 28  through  40) 
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The  third  source  is  an  attempt  at  a  scientific  test  of  message  feedback 
effectiveness  in  AARs.  Downs  et  al.  performed  message  content  analysis  on  six  battalion 
staff  AARs  to  evaluate  the  adherence  to  principles  of  effective  feedback.  The  AARs  were 
conducted  as  part  of  command  and  staff  training  with  the  Army  Training  Battle 
Simulation  System  (ARTBASS).  The  content,  form  (question  or  comment),  level  of 
specificity/detail,  and  valence  (positive/negative  nature)  of  verbal  messages  uttered 
during  the  AARs  were  measured. 

Downs  et  al.  conclude  that  seven  components  are  necessary  for  an  effective  AAR. 
These  are  introduction,  topics,  participation,  description,  evaluation,  goals,  and  summary. 
(Downs  et  al.,  1987,  B-5)  The  first  is  the  introduction  given  by  the  AAR  leader.  The 
introduction  is  “very”  important  in  setting  a  participatory  climate.  Downs  et  al.  state  that 
the  climate  should  be  “structured  but  not  intimidating.”  The  research  team  does  not 
recommend  how  to  achieve  this  climate  other  than  saying  that  the  AAR  leader  should 
have  relaxed,  engaging,  and  unthreatening  mannerisms.  To  establish  structure  they 
recommend  reviewing  the  “learning  agenda”  for  the  exercise  and  setting  the  expectancy 
of  the  AAR  participants  for  the  roles  they  are  to  play  during  the  AAR.  (Downs  et  al., 
1987,  B-1  to  B-2  and  B-5)  Downs  et  al.  do  not  define  what  a  learning  agenda  is.  They 
do  mention  that  the  learning  objectives  are  a  component  of  the  learning  agenda.  (Downs 
et  al.,  1987,  B-2)  However,  an  example  or  definition  is  not  given. 

Bosley  et  al.  agree  that  the  introduction  is  important  to  the  AAR.  They  state  that 
the  trainees  should  sense  a  climate  of  open  discussion  from  the  introduction.  The  unit 
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should  know  why  they  are  there,  what  to  expect,  and  what  their  role  in  the  process  is. 
(Bosley  et  al.,  1979,  15  and  16)  Bosley  et  al.  and  Word  also  recommend  stressing  the 
role  of  the  AAR  leader  as  a  moderator/facilitator.  (Bosley  et  al.,  1979, 16,  and  Word, 

1987, 34) 

The  second  component  of  effectiveness  is  the  AAR  topics  to  be  discussed.  (Downs 
et  al.,  1 987,  B-2)  The  research  team  identifies  the  six  most  common  topics  discussed  but 
fail  to  say  why  they  are  important  to  effectiveness.  In  their  executive  summary,  the  team 
suggests  that  the  most  common  AAR  topics  be  the  source  of  future  training  exercise 
objectives  and  AAR  discussion  points.  (Downs  et  al.,  1987,  viii)  In  any  case.  Downs  et  al. 
are  in  direct  conflict  with  Bosley  et  al.  and  Word.  Both  of  them  say  that  the  topics  or 
performance  problems  cannot  be  predetermined  and  that  the  participants,  not  the  AAR 
leader,  must  discover  them.  (Bosley  et  al.,  1979, 14,  and  Word,  1987, 29  to  30  and  33) 
Bosley  et  al.  and  Word  realized  that  every  unit  in  the  Army  has  different  strengths  and 
weaknesses  at  a  given  period  in  time.  A  much  larger  and  more  representative  sample  of 
vuiits  must  be  used  to  determine  common  problem  trends.  Even  when  identified,  these 
trends  are  only  valid  for  the  factors  (focus  of  training,  prior  proficiency  levels,  training 
resources  used,  personnel  turbulence,  etc.)  that  defined  the  training  environment  at  that 
specific  point  in  time. 

Downs  et  al.  identify  participation  as  an  “extremely”  important  component  of  an 
effective  AAR  that  is  directly  related  to  the  atmosphere  created  during  the  introduction. 
(Downs  et  al.,  1987,  B-2)  Although  they  do  not  offer  data  to  support  this  conclusion,  it 
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appears  logical  and  SHERIKON  (1996)  did  find  research  to  support  it.  Downs  et  al.  do 
tie  unit  participation  to  the  use  of  “why”  (leading)  questions  by  the  AAR  leader.  (Downs 
et  al.,  1987, 16) 

Downs  et  al.  tracked  the  number  and  source  (participant)  of  each  utterance,  to 
include  the  AAR  leader.  Hence,  conclusions  about  the  amount  of  participation  that 
occurred  during  the  AAR  were  drawn  from  the  data.  Their  participation  finding  and 
conclusion  are  summarized  below. 

The  number  of  participants  and  their  degree  of  participation  was  highly 
influenced  by  the  individuals  leading  the  AAR  sessions  as  can  be  seen 
from  contrasting  two  battalions.  Of  all  six  AAR  sessions,  the  AAR  leader 
of  Battalion  2  asked  the  most  [leading  and  open-ended]  questions  of  his 
personnel  and  had  the  highest  number  of  participants.  Conversely,  the 
AAR  leader  of  Battalion  4  was  among  the  three  AAR  leaders  who  asked 
the  fewest  number  of  [leading  and  open-ended]  questions  and  had  the 
lowest  number  of  participants.  Feedback  should  come  from  multiple 
sources  as  opposed  to  a  single  source.  Participation  in  the  AAR  also 
leads  to  greater  acceptance  of  evaluative  feedback.  (Downs  et  al.,  1987, 

21)'' 


While  the  conclusion  (italics)  seems  obvious,  the  findings  are  a  bit  misleading. 
Since  the  study  did  not  allow  for  the  AAR  leaders  to  conduct  multiple  AARs,  the  AAR 
leader  is  not  a  controlled  variable.  Therefore,  the  certainty  with  which  the  level  of  trainee 
participation  can  be  attributed  to  each  of  the  AAR  leaders  is  questionable. 

Comparing  the  actual  data  of  the  two  AAR  leaders  renders  the  justification  for 
Downs  et  al.’s  second  statement  above.  The  2“'*  battalion’s  AAR  leader  asked  199 
questions  and  declared  288  comments.  The  4*"  battalion’s  AAR  leader  asked  23 
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questions  and  declared  270  comments.  (Downs  et  al.,  1987, 19)  There  was  a  large 
variation  between  the  two  AAR  leaders. 

Combining  the  data  of  all  six  AARs  indicates  that  there  was  a  failure  across  the 
board  to  implement  the  questioning  technique  to  prompt  participation.  In  all,  AAR 
leaders  asked  354  questions  and  declared  1,959  comments.  Moreover,  Downs  et  al.’s 
analysis  found  that  AAR  leaders  employed  leading  questions  just  15%  of  the  time. 
(Downs  et  al.,  1987,  19  through  23)  Thus,  the  2°^  battalion  AAR  leader’s  performance 
was  an  exception  rather  than  the  norm. 

Comparing  the  percentage  of  utterances  by  participant  reveals  a  clearer  picture  of 
the  amount  of  trainee  participation  that  actually  occurred.  For  instance,  consider  die 
AAR  in  which  the  highest  rate  of  participation  was  observed  -  the  2”‘‘  battalion’s  AAR. 
In  this  AAR,  the  AAR  leader,  OPFOR,  and  unidentified  sources  were  responsible  for 
1 1 .3%,  6.2%,  and  1 .7%  of  the  utterances  respectively.  Of  the  remaining  80.8%  of  the 
utterances,  the  battalion  and  company  commanders,  4  of  the  20  unit  participants,  were 
responsible  for  65.1%.  (Downs  et  al.,  1987, 1 1)  Thus,  counting  the  AAR  leader’s 
questions,  more  than  three  quarters  of  the  trainees  were  involved  in  less  than  16%  of  the 
total  discussion. 

Combining  all  six  AARs,  the  source  and  percentage  of  the  utterances  are 
summarized  in  Table  2.1,  Percentage  of  Utterances  by  Source  for  Six  AARs.  (Downs  et 


^  “Source”  and  “sources”  here  refer  to  individuals  attending  the  AAR. 
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Table  2.1,  Percentage  of  Utterances  by  Source  for  Six  AARs 


Source 

%  Utterances 

AAR  Leader 

17.1 

OPFOR 

7.3 

Trainer 

6.9 

Unknown 

2.6 

Trainees 

66.1 

Commanders 

53.7 

Remainder  of  Unit 

12.4 

al.,  1987,  1 1)  The  poor  distribution  of  participatory  remarks  by  the  unit  can  probably  be 
attributed  to  a  couple  of  factors.  The  AAR  leaders’  inexperience  or  incompetence  at 
distributing  questions  amongst  the  group  and/or  the  authority  of  the  unit  commanders 
intimidating  other  participants  and  AAR  leaders. 

They  also  emphasize  that  participation  is  dependent  upon  the  discussion  topic  of 
individual  and  group  performance  and  the  trainees’  perception  that  it  is  permissible  to 
disagree.  (Downs  et  al.,  1987,  B-2)  They  offer  no  data  nor  previous  psychological  studies 
as  evidence  of  permissive  environments  facilitating  participation  or  that  people  are  more 
motivated  to  speak  about  performance  in  feedback  contexts.  These  appear  to  be  logical 
presumptions  that  are  verified  by  the  experiment  and  agreed  with  by  both  Bosley  et  al.. 
Word,  and  SHERIKON’s  research. 
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The  third,  fourth,  and  fifth  components  of  an  effective  AAR  concern  the 
characteristics  of  the  message  itself  Description,  evaluation,  and  goals  are  required 
elements  of  a  message  for  it  to  be  effective  feedback.  Downs  et  al.  cite  a  number  of 
psychological  research  studies  for  this  conclusion.  (Downs  et  al.,  1987, 5)  Locke, 
Frederick,  Cousins,  and  Bobko  (1983),  Kim  and  Hamner  (1976),  and  Nemeroff  and 
Cosentino  (1979)  all  conducted  studies  that  showed  that  messages  that  reference  goals  in 
conjimction  with  messages  that  provide  feedback  about  specific  performance  enhance  the 
feedback  process  and  improve  subsequent  performance.  Nadler  (1979)  and  Cusella 
(1987)  indicate  feedback  that  evaluates  an  action  as  right  or  wrong  improves  performance 
more  than  feedback  without  any  evaluative  information.  There  is  an  inherent  problem  in 
requiring  verbal  feedback  messages  to  contain  these  elements.  It  takes  education  and 
training  to  achieve  proficiency  in  transmitting  these  types  of  messages.  The  natural 
course  of  action  would  be  to  give  AAR  leaders  the  feedback  training.  However, 
implementing  this  may  cause  them  to  move  back  toward  lecturing  the  trainees  to  provide 
“effective  feedback.” 

Word  (1987)  and  subject  matter  experience  agree  that  the  solution  is  for  AAR 
leaders  to  guide  and  focus  the  discussion  to  achieve  the  desired  characteristics.  This 
would  mean  specifically  identifying  the  strength/weakness,  the  good/bad  behavior  or 
action  that  was  its  cause,  and  the  corrective  action  that  will  improve  performance  and 
achieve  the  desired  result  -  the  goal.  (Word,  1987, 34  to  36) 
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These  message  characteristics  can  also  apply  to  non-verbal  feedback.  Non-verbal 
feedback  consists  of  audio  and  video  replay  of  performance,  charts  and  sketches  depicting 
BDA  and  instances  of  the  exercise,  and  various  statistical  and  text  performance 
summaries.  Word  states  that  AAR  leaders  must  carefully  consider  the  use  of  each  AAR 
aid  to  ensure  that  the  desired  feedback  point  is  made  clearly  and  effectively.  (Word,  1987, 
35  and  38  to  39)  This  is  in  line  with  Downs  et  al.’s  requirement  for  the  message  to  be 
specific  in  description,  objective  enough  to  measure,  and  focused  on  a  learning  point. 

Downs  et  al.  advocate  that  the  last  component  of  an  effective  AAR  is  the 
conclusion.  They  state  that  the  conclusion  must  highlight  the  main  points  of  the  AAR 
and  link  them  to  AAR  “learning  objectives”  and  training  objectives.  (Downs  et  al.,  1987, 
B-5)  Bosley  et  al.  and  Word  both  agree  with  this  finding.  (Bosley  et  al.,  1979, 19,  and 
Word,  1987,  36) 

The  final  source.  Analysis  of  AVCATT  After  Action  Review  Requirements,  was 
a  research  effort  conducted  by  SHERIKON  Inc.  Its  purpose  was  to  identify  AAR 
requirements  for  the  Aviation  Combined  Arms  Tactical  Trainer  (AVCATT)  relative  to 
the  Close  Combat  Tactical  Trainer  (CCTT)  AAR  system  and  AAR  effectiveness  research. 
AVCATT  is  a  virtual  simulation  training  system  that  will  integrate  aviation  crews  in 
support  of  maneuver  elements  in  CCTT. 

The  SHERIKON  research  team  conducted  a  literature  review  to  determine  the 
requirements  for  an  effective  AAR.  However,  they  did  not  include  the  work  of  Bosley  et 
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al.,  Word,  nor  Downs  et  al.  They  did  cover  the  work  of  research  psychologists  to 
determine  if  performance  feedback  is  important.  These  were  Alexander,  Kepner,  & 
Tregoe  (1962),  Bessemer,  Shlechter,  Nesselroade,  &  Anthony  (1995),  Hankinson  (1987), 
and  Holmes  (1976).  SHERIKON  also  reviewed  the  work  of  Briggs  &  Johnston  (1966), 
Cleveland  (1985),  Cleveland  &  McGill  (1985),  Horrocks  (1960),  Legge,  Gu,  &  Luebker 
(1991),  Meister  (1976),  and  Rosenberg  (1960)  to  conclude  how  feedback  should  be 
presented  and  why  it  is  effective.  (SHERIKON,  1996b,  4  to  8  and  34  to  36) 

SHERIKON  concluded  that  there  were  seven  elements  that  made  an  AAR 
effective.  These  elements  are:  1)  feedback  should  only  consist  of  performance  outcome 
data  not  apparent  nor  known  to  the  trainees;  2)  feedback  should  be  specific  in  judgment; 
3)  timeliness  of  the  corrective  training;  4)  feedback  should  be  visually  presented  rather 
than  verbally;  5)  the  training  objectives  and  plans  of  execution  must  be  clearly 
articulated;  6)  trainees  must  receive  a  summary  of  critical  actions  and  events;  and  7) 
trainee  participation  in  the  discussion. 

First,  trainees  require  a  debrief  of  the  actions  and  their  effects  that  cannot  be 
readily  detected  during  die  exercise.  (SHERIKON,  1996b,  34)  In  other  words,  trainees 
require  information  to  fully  understand  the  cause  and  effect  relationships  of  what 
happened  in  order  to  make  behavioral  changes  that  affect  performance.  It  is 
SHERIKON’ s  position  that  data  should  be  provided  only  for  the  cause  and  effect 
relationships  that  are  not  readily  apparent  to  participants  during  the  training. 
(SHERIKON,  1996b,  7)  They  view  AAR  performance  feedback  as  redundant  when 


37 


participants  can  detect  or  see  the  eonsequences  of  their  actions  during  training. 
(SHERIKON,  1996b,  5)  They  argue  that  participants  should  already  have  the  feedback 
they  need  to  modify  behavior, 

Bosley  et  al  and  Downs  et  al.  do  not  address  what  feedback  should  be  presented. 
Word  states;  “What  we  discover  in  the  mission  is  just  some  obvious  lessons  learned. 

The  real  critical  learning  will  surface  in  the  AAR.”  (Word,  1987, 32)  Subject  matter 
experience  conflicts  with  SHERIKON’s  conclusion  since  each  individual  is  left  to  his 
own  interpretation  of  what  happened  and  why.  Many  training  exercises  are  long  and 
complex  enough  for  participants  to  forget  what  they  did.  In  this  case,  they  need  the 
feedback  to  refresh  memory.  Everyone  benefits  from  understanding  the  value  of  cause 
and  effect  relationships  with  respect  to  other  actions  and  the  overall  results  of  unit 
performance.  The  actions  whose  consequences  are  visible  are  often  linked  to  the  actions 
whose  consequences  are  not  and  vice  versa.  To  establish  the  relationships,  it  is  often 
necessary  to  link  the  feedback.  Also,  individuals  remember  an  action  differently  and  may 
not  realize  all  of  the  consequences  of  their  actions.  Furthermore,  an  individual/crew’s 
perception  of  a  cause  and  effect  relationship  is  exactly  what  is  supposed  to  be  discussed 
to  determine  what  happened  and  why.  They  must  be  able  to  completely  reconstruct  this 
for  other  participants  to  benefit.  In  other  words,  feedback  on  actions  whose  consequences 
are  apparent  will  positively  or  negatively  reinforce  others  who  perform  the  same  action. 
They  can  relate  the  feedback  to  their  own  performance. 
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The  impetus  for  SHERIKON’s  conclusion  concerning  feedback  was  a  study  by 
Alexander  et  al.  (1962)  of  Air  Force  air  defense  operator  teams  detecting,  identifying, 
tracking,  and  targeting  aircraft.  Alexander  et  al.  found  that  the  AAR  increased  overall 
average  performance  gains  by  a  magnitude  of  1 8  times.  But  those  team  functions  that 
yielded  immediate  and  observable  (apparent)  information  during  their  performance  were 
less  affected  by  the  AAR  feedback  than  were  the  functions  that  yielded  less  intrinsic  or 
observable  information  to  the  trainee.  (Alexander  et  al.,  1962, 208  and  210)  In  other 
words,  when  the  trainees  could  immediately  observe  the  effect  of  their  actions,  they  were 
able  to  correct  themselves  and  did  not  attend  to  the  AAR  feedback.  When  trainees  were 
not  aware  of  the  performance  outcome  of  their  actions,  the  AAR  feedback  (for  those 
functions)  had  an  effect  on  performance.  However,  Alexander  et  al.  did  not  consider  nor 
measure  the  effect  of  feedback  for  functions  with  observable  consequences  on  other 
individuals  who  had  to  perform  the  same  function.  Thus,  he  could  not  draw  conclusions 
about  the  benefit  gained  from  watching  and  learning  how  others  perform  the  same  task. 
While  the  performance  feedback  in  the  AAR  may  be  uninteresting  and  obvious  to  one 
trainee,  it  may  benefit  another. 

The  second  element  of  effectiveness  that  SHERIKON  discusses  is  that  feedback 
must  be  specific  rather  than  qualitative.  (SHERIKON,  1996b,  34)  This  is  so  trainee 
discussion  does  not  focus  on  the  subjective  qualities  of  the  feedback.  This  is  what  Word 
advocated  and  Downs  et  al.  argued  feedback  message  content  should  be  with  their 
experiment.  However,  SHERIKON  qualifies  this  finding  with  the  results  of  Meister 
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(1 976).  The  amount  of  feedback  detail  required  by  the  trainees  is  determined  by  their 
collective  skill  level.  Thus,  the  detail  of  feedback  varies  directly  with  team  skill;  more 
skill  requires  more  feedback  detail.  SHERIKON  points  out  that  Meister  finds  the 
converse  is  also  true  —  too  much  detail  will  degrade  feedback  effectiveness  and 
subsequent  performance.  (SHERIKON,  1996b,  6) 

Even  though  none  of  the  other  sources  address  this  issue,  personal  experience  and 
cognitive  research  confirms  Meister’s  finding  that  high  performing  units  require  more 
detail  in  performance  feedback.  People  solve  problems  (learn)  iteratively  by  beginmng 
with  an  analogy  to  past  experience.  (Anderson,  1993,  Newell,  1992,  and  Simon  & 

Newell,  1972)  Each  iterative  step  requires  a  specific  amount  of  knowledge  units  and  too 
many  extraneous  knowledge  units  will  obscure  those  needed.*  Indeed,  poorly  trained 
units  cannot  process  nor  even  understand  the  relevance  of  feedback  if  it  is  too  detailed. 

As  Meister  suggests,  the  unit,  if  left  to  their  own  devices,  will  probably  try  to  process  all 
the  details  and  fail  to  identify  problems  and  solutions  commensurate  with  the  unit’s  skill 
level.  That  is,  if  the  AAR  leader  lets  the  vmit  proceed  down  that  path.  Subject  matter 
experience  indicates  that  it  is  easy  to  focus  the  trainees  on  a  few  salient  performance 
outcomes  that  will  lead  them  to  problems  and  discussion  appropriate  to  their  skill  level. 
This  is  accomplished  by  simply  telling  the  trainees  to  ignore  the  undesirable  feedback  and 
saying;  “Let’s  just  focus  on  this  (result)  relative  to  that  action.”  This  takes  their  attention 


’  This  process  is  explained  in  greater  detail  in  Appendix  A,  Theoretical  Foundations. 
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off  the  intimidating  detail.  In  fact,  the  group  is  usually  glad  to  ignore  some  detail  in  the 
hope  that  they  may  find  evidence  of  strong  performance. 

SHERIKON’s  third  element  of  effectiveness  is  the  amount  of  time  that  elapses 
from  the  point  of  learning  a  lesson  in  the  AAR  to  the  point  when  the  unit  can  practice  or 
train  to  implement  the  lesson.  (SHERIKON,  1996b,  34  to  35)  The  unit  must  have 
sufficient  time  to  reinforce  the  lessons  with  correct  behavior  and  rehearsal  in  order  to 
fully  learn  from  the  feedback.  Otherwise,  the  trainees  will  not  fully  retain  the  learning 
point  and  its  application.®  (SHERIKON,  1996b,  6)  This  is  an  excellent  point  as  it  takes 
the  linking  of  AAR  points  to  training  objectives,  that  Word  and  Downs  et  al.  emphasize, 
one  step  further.  For  SHERIKON’s  AAR  process  then,  the  next  step  in  the  doctrinal 
training  cycle  would  become  the  last  step  of  the  AAR.  The  benefit  of  correcting  specific 
weaknesses  immediately  is  obvious  and  also  proven  by  Meister’s  research.  (Meister, 
1976)  The  only  obstacle  would  be  having  the  required  training  resources,  including  time, 
to  train  specific  tasks  under  specific  conditions. 

The  fourth  effectiveness  finding  is  that  visual  feedback  is  superior  to  verbal 
feedback.  (SHERIKON,  1996b,  35)  SHERIKON  cites  research  done  by  Briggs  and 
Johnston  (1966)  to  substantiate  this  and  gives  two  reasons  for  it.  The  first  is  the  AAR 
leader’s  inability  to  recognize  and  commimicate  the  items  important  to  the  individuals 
and  unit  that  will  lead  to  improved  performance.  This  lends  weight  to  Bosley  et  al.’s  and 


‘  Empirical  evidence  for  this  statement  is  given  in  Anderson  (1993)  and  covered  later  in  this  chapter  and 
Appendix  A,  Theoretical  Foundations. 
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Word’s  defiance  of  Downs  et  al.’s  presumption  that  AAR  topics  can  be  predicted. 
(Downs  et  al.,  1987,  B-2) 

The  second  is  that  a  picture  is  worth  a  thousand  words.  SHERIKON  references 
the  work  of  Cleveland  (1985)  and  Cleveland  and  McGill  (1985)  that  shows  graphic 
displays  communicate  information  more  efficiently  than  numerical  tables.  This  was 
reinforced  with  the  experiments  of  Legge,  Gu,  and  Luebker  (1991)  who  foxmd  that 
performance  with  scatter  plots  showed  clear  evidence  of  parallel  processing  compared  to 
that  of  the  numerical  tables  which  appeared  to  be  processed  in  a  serial  manner  at  a  fixed 
rate.  Thus,  the  human  visual  system  seems  to  have  the  ability  “to  process  global  pattern 
features  at  a  glance”  while  the  aural  system  processes  items  sequentially  at  a  slower  rate. 
(SHERIKON,  1996b,  6  to  7)  Word  confirms  this  in  his  paper.  For  example,  he  states 
that  the  best  way  to  point  out  a  navigational  problem  to  someone  is  to  “show  his  unit 
steadily  moving  out  of  sector  on  that  [plan  view]  screen.”  (Word,  1987,  39)  A  “plan 
view”  is  a  birds-eye,  two  dimensional  view  of  an  action,  event,  or  situation.  It  is  usually 
depicted  on  a  large  scale  map. 

SHERIKON’ s  fifth  element  is  that  the  training  objectives  and  fiiendly  and 
OPFOR’s  plans  of  action  must  be  clearly  understood  by  the  participants.  (SHERIKON, 
1996b,  35)  This  understanding  provides  a  framework  for  the  unit  to  identify  critical 
behaviors  in  terms  of  what  was  supposed  to  happen  and  then  compare  them  to  what 
actually  happened.  (SHERIKON,  1996b,  35)  This  ensures  that  participants  take  into 
account  all  factors  and  conditions  that  influence  an  identified  strength  or  weakness.  It 
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allows  accurate  diagnosis  of  why  a  set  of  actions  caused  a  specifie  outcome.  Bosley  et 
al.,  Word,  and  subjeet  matter  experience  verify  this  important  element.  (Bosley  et  al., 
1979, 6  and  8,  and  Word,  1987, 32  to  34)  Downs  et  al.  also  support  this  finding  but 
spend  more  effort  emphasizing  the  AAR  leader’s  questioning  technique  to  aehieve  a  clear 
understanding  of  the  training  conditions. 

The  sixth  element  is  that  the  critieal  events  and  turning  points  of  the  training  must 
be  summarized  and  presented  to  the  participants  to  aid  their  recollection  of  their  own 
actions.  (SHERIKON,  1996b,  35)  This  element  of  effeetiveness  is  important  to  defining 
the  cause  and  effect  relationships  of  trainee  aetions  and  battlefield  results.  SHERIKON 
identifies  this  as  the  starting  point  for  trainee  discovery  of  why  something  happened  and 
how  to  do  it  better.  (SHERIKON,  1996b,  36)  This  element  should  not  be  confused  with 
the  first,  the  requirement  to  provide  feedback  for  actions  whose  outcomes  are  not 
apparent  to  the  trainees.  The  first  element  refers  to  providing  performance  outcomes 
(results)  for  speeifie  actions.  These  specific  actions  may  or  may  not  be  part  of  a  critical 
event/tuming  point  in  the  training  exercise. 

The  training  objeetives  eombined  with  the  plans  of  action  define  the  goal  the  unit 
was  trying  to  aehieve.  This  eombination  is  an  effieient  method  of  focusing  discussion 
upon  a  problem  area  where  participants  can  clearly  define  cause  and  effect  relationships. 
Bosley  et  al.  and  Downs  et  al.  both  agree  with  this  finding.  (Bosley  et  al.,  1979, 15,  and 
Downs  et  al.,  1987,  B-5) 
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Word  agrees  but  also  points  out  it  is  only  one  method  of  beginning  or  guiding  a 
discussion.  An  alternative  is  to  compare  what  the  unit  was  trying  to  achieve  to  a  doctrinal 
solution.  Still  another  method  is  to  use  the  BDA  or  other  objective  performance 
measures  as  a  starting  point.  (Word,  1987, 30  and  35)  Subject  matter  experience  has 
indicated  that  these  methods  are  mutually  inclusive.  For  example,  starting  with  the  BDA 
(the  performance  outcome),  the  discussion  of  what  happened  and  why  will  logically  lead 
to  an  examination  of  the  planned  actions  and  then  a  comparison  to  a  doctrinal  solution. 

Or,  a  doctrinal  solution  can  be  compared  to  the  plan  of  action  initially  and  then  evaluated 
with  respect  to  the  performance  outcome.  Either  way,  the  doctrinal  solution  is  always 
discussed  to  determine  the  best  solution  to  the  problem. 

The  last  element  of  effectiveness  is  trainee  participation  in  the  discussion  process. 
(SHERIKON,  1996b,  36)  SHERIKON  echoes  Bosley  et  al..  Word,  and  Downs  et  al.  as 
they  point  out  that  active  participation  is  necessary  to  provide  differing  perspectives  of 
the  topics,  promote  the  generalization  of  the  lessons  learned  to  a  wider  range  of 
situations,  and  increase  participant  motivation  which  increases  discussion  contribution 
and  quality.  (SHERIKON,  1996b,  36)  SHERIKON  also  points  out  that  it  is  the  AAR 
leader’s  responsibility  to  guide  and  focus  the  discussion  on  the  points  that  are  important 
and  meaningfiil  with  respect  to  patterns  of  effective  and  ineffective  behavior. 
(SHERIKON,  1996b,  36) 

SHERIKON’ s  research  confirms  that  learning  takes  place  during  participatory 
discussion  or,  in  their  words,  a  “debriefing  session.”  This  combined  with  the  finding  that 


44 


the  learning  must  be  reinforced  with  correct  behavior  as  soon  as  possible  after  the 
discussion  seem  to  be  the  salient  points  of  their  six  elements  of  effectiveness.  Subject 
matter  experience  supports  this  position. 


Effectiveness  Synthesized 

The  term  “effectiveness”  is  defined  as  “having  the  intended  or  expected  effect; 
serving  the  purpose.”  (Morris,  W.,  The  American  Heritage  Dictionary.  1980)  Thus,  the 
effectiveness  of  the  AAR  is  determined  by  improvement  in  individual  and  unit 
performance.  Plainly,  this  improvement  is  dependent  upon  how  well  the  unit  can  learn 
from  the  training  experience  during  the  AAR.  In  turn,  how  well  a  unit  learns  depends 
upon  the  acceptance  or  recognition  of  the  learning  point  by  the  group  and  how  well  they 
understand/relate  the  lesson  to  their  own  experience. 

AAR  effectiveness,  as  shown  by  the  research  literature,  consists  of  four  elements. 

•  Participation  in  the  discussion; 

•  Discussion  focus  on  learning  point(s)  -  problem  and  solution; 

•  Learning  reinforcement  -  retraining; 

•  Time  -  delay  before  the  AAR,  length  of  AAR,  and  delay  before  retraining. 

All  of  these  factors  were  directly  identified  by  Bosley  et  al..  Word,  Downs  et  al.,  and 
SHERIKON  and  are  verified  by  subject  matter  experience.  The  other  elements 
emphasized  by  the  sources  are  supporting  actions,  techniques,  or  requirements  to  achieve 
one  or  more  of  the  above  factors. 
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Discussion  Participation 


Active  trainee  participation  in  the  AAR  was  identified  as  the  most  important 
element  of  effectiveness  by  all  researchers.  Participation  in  the  AAR  is  best  when 
complete.  Each  member  of  the  training  unit,  the  OPFOR,  the  unit  trainer,  and  the  AAR 
leader  must  be  included.  This  will  ensure  a  number  of  results  that  will  contribute  directly 
to  the  effectiveness  of  the  AAR.  First,  it  defines  how  good  the  second  factor,  the  learning 
points,  are.  Full  participation  allows  the  “what  happened”  and  “why  it  happened”  to  be 
completely  and  accurately  reeonstructed.  With  100%  information,  cause  and  effect 
relationships  between  behaviors  and  results  are  better  established.  Once  the  problem  is 
more  fully  understood  in  these  terms,  partieipants  lend  multiple  viewpoints  and 
considerations  to  solve  the  problem.  Participation  increases  the  accuracy  of  problem 
interpretation  such  that  all  facets  of  the  problem  are  realized,  considered,  and  accoimted 
for.  Finally,  participation  in  solving  a  problem  and  then  developing  corrective  measures 
motivates  trainees  to  participate  more  and  leaders  to  implement  the  solution. 

Behavioral  psychologists  outside  of  ARI  have  verified  the  importance  of  full 
participation  in  group  problem  solving  research.  Hollenbeck  et  al.  (1995)  tested  a  theory 
of  decision  making  performance  for  hierarchical  teams  with  distributed  expertise.  These 
teams  were  designed  to  model  military  teams  in  which  there  are  status  differences  among 
members,  members  have  differing  levels  of  expertise  and  knowledge,  and  one  member  is 
held  primarily  responsible  for  the  decision.  Hollenbeck  et  al.  found  that  problem  solving 
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efficiency  and  accuracy  increased  when  information  was  collected  and  distributed  to  all 
team  members  and  when  all  members  participated.  (Hollenbeck  et  al.,  1995, 292  to  312) 

In  1959,  Smith  and  Kight  conducted  a  study  on  different  effects  on  problem 
solving  efficiency  in  training  groups.  Their  findings  show  that  the  AAR  leader  can  have 
some  affect  on  participation.  Their  data  showed  that  personalized  feedback  for  group 
members  “markedly,  and  consistently,  improved  group  problem  solving  efficiency.” 
(Smith  and  Kight,  1959, 21 1)  Hence,  if  the  AAR  leader  can  verbally  laud  a  soldier  for 
contributing  to  the  discussion,  he  and  the  other  unit  members  will  be  more  likely  to 
participate.  This  responsibility  will  require  a  sizable  portion  of  the  AAR  leader’s 
attention.  However,  if  the  AAR  leader  is  conscientiously  providing  participation 
feedback,  then  he  is  also  not  lecturing. 

Participation  also  breaks  down  the  group’s  natural  resistance  to  criticism.  When 
team  members  are  aware  of  the  problem  and  solution,  each  member  will  work  to  facilitate 
the  solution  implementation.  Leaders  responsible  for  implementation  will  perceive  the 
pressure  to  do  so  from  superiors,  peers,  and  subordinates.  This  contributes  to  an 
organizational  enviromnent  in  the  Army  that  pressures  leaders  to  correct  a  problem  when 
it  is  identified  without  waiting  to  be  told  to  do  so.  (Word,  1987,  33,  and  personal 
experience)  Hence,  participation  ensures  that  the  problem  is  fully  recognized,  alternative 
solutions  considered,  and  the  corrective  action  implemented. 

Two  components  that  researchers  have  identified  affect  participation.  They  are 
the  AAR  leader’s  ability  to  establish  an  open,  non-threatening  climate  that  encourages 
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participation  and  his  use  of  leading  and  open-ended  questions.  As  Bosley  et  al.,  Word, 
and  Downs  et  al.  point  out,  the  trainee’s  perception  that  it  is  permissible  to  disagree  is 
important  to  the  climate.  Additionally,  they  emphasized  the  use  of  leading  and  open- 
ended  questions  to  prompt  participation  in  the  discussion.  This  questioning  technique 
can  be  used  to  prompt  individuals  to  contribute  if  they  are  shy  or  to  ensure  that  a  lesson  is 
fully  articulated. 

Another  key  element  of  participation  is  the  mechanism  that  the  AAR  leader  uses 
to  guide  discussion  so  that  time  and  effort  are  not  wasted  on  ineffectual  events.  This 
mechanism  consists  of  purpose  and  objectives  and  a  format  for  the  AAR.  The  result  of 
applying  these  discussion  guidance  mechanisms  is  that  discussion  topics  on  how  to 
improve  performance  are  quickly  identified  and  focused.  Tangential  and  extraneous 
topics  are  minimized  and  discussion  focus  is  sought  during  the  AAR. 


Discussion  Focus 

The  discussion  is  the  principal  learning  medium  in  the  AAR.  All  researchers 
highlighted  the  discussion’s  components  as  critical  to  effectiveness.  They  agree  that  the 
discussion  must  establish  what  happened,  why  it  happened,  and  how  to  do  it  better. 
Indeed,  this  is  die  learning  point  of  an  AAR.  To  achieve  a  learning  point,  discussion 
must  result  in  a  clear  articulation  of  what  happened  and  why.  The  training  conditions,  as 
well  as  a  plan  to  reinforce  or  correct  behavior  with  respect  to  those  conditions  must  be 
discussed,  considered,  and  re-discussed.  In  accordance  with  doctrine,  the  plan  must 
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consist  of  the  tasks  to  be  trained,  conditions  in  which  to  perform  those  tasks,  and 
standards  of  task  performance.  Finally,  this  plan  must  be  practical  in  that  it  is  within  the 
capabilities  of  the  unit  to  execute. 

The  number  of  learning  points  articulated  in  an  AAR  does  not  necessarily  bear  on 
its  effectiveness.  This  is  because  just  one  realized  learning  point  will  improve 
performance  in  some  manner.  The  unit  skill  level,  their  collective  fatigue  level  and 
mood,  and  their  perception  of  the  utility  of  the  training  will  affect  the  number  of  learning 
points  they  achieve.  Additional  influences  are  the  complexity  of  the  training;  the  AAR 
leader’s  skill  at  guiding,  moderating,  and  facilitating  discussion;  and  the  clarity  and  detail 
of  the  performance  feedback.  All  of  these  interrelate  to  determine  how  many  learning 
points  are  produced. 

To  define  a  learning  point,  the  AAR  leader  must  guide  and  focus  the  discussion. 
Bosley  et  al..  Word,  Downs  et  al.,  and  SHERIKON  stated  that  an  efficient  method  of 
doing  this  was  to  ensure  that  the  training  objectives  and  conditions  were  clearly 
understood  to  serve  as  a  framework  for  the  discussion.  Word  and  SHERIKON  then 
recommend  beginning  the  discussion  with  objective  performance  results  such  as  the 
casualty  exchange  ratio  or  time  required  to  complete  a  time  sensitive  task.  The  search  for 
the  cause  of  these  effects  will  reveal  the  critical  actions  and  events  that  must  be 
investigated.  A  detailed  cause  and  effect  analysis  will  lead  to  the  appropriate  corrective 
action  or  training.  Articulation  of  the  corrective  action  or  training  plan  is  the  point  of  the 
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discussion.  The  plan  must  address  the  purpose  of  training  and  link  the  original  training 
objectives  to  the  objectives  of  the  corrective  training. 

The  purpose  of  the  AAR  is  to  improve  individual  and  collective  performance;  this 
never  changes.  The  objectives  of  the  AAR  are  to:  1)  define  the  problems  that  will 
improve  performanee,  2)  articulate  the  solution  in  relation  to  the  training  conditions 
experienced  to  increase  learning,  and  3)  translate  the  solutions  into  future  training 
objeetives  to  reinforce  learning.  The  AAR’s  purpose  and  objectives  are  different  from 
the  training  purpose  and  objeetives.  However,  problems  and  solutions  identified  in  an 
AAR  should  support  the  purpose  and  objectives  of  training  in  that  they  coneem  behaviors 
that  are  associated  with  the  specified  training  tasks.  This  will  also  limit  the  number  of 
discussion  points. 

The  proeedure  to  achieve  the  AAR  objectives  must  be  rigidly  enforced.  The  most 
efficient  discussion  focuses  on  what  happened,  why  it  happened,  and  how  it  can  be  done 
better.  Discussion  of  other  topics  or  events  should  not  be  permitted  in  order  to  conserve 
time  and  maintain  the  focus  of  the  trainees’  attention. 

The  AAR  must  allow  the  unit  to  clearly  understand  situational  problems  and 
realize  practical  solutions.  Therefore,  the  conditions  under  which  training  is  performed 
must  be  related  to  the  behaviors  exeeuted  to  aehieve  specified  goals.  Once  the  faetors 
that  influence  a  weakness  are  identified  and  the  cause  and  effect  relationships  established, 
solutions  must  be  identified.  The  solution  is  “practical”  only  if  the  unit  has  the  resources 
and  capabilities  necessary  to  implement  it. 
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Additionally,  imit  strengths  and  weaknesses  must  be  identified  and  assessed  with 
respect  to  past  performance/experience  and  Army  performance  standards  and  doctrine. 
The  AAR  leader  must  aid  the  trainees  in  the  identification  of  strengths  and  weaknesses  in 
an  objective  manner.  Unless  intimately  familiar  with  the  unit’s  past  performance  and  a 
member  of  the  unit’s  chain  of  command,  he  must  avoid  criticizing  the  group  directly  and 
leave  the  performance  assessment  to  the  unit  and  its  leaders.  This  avoids  the  negativism 
associated  with  the  judgments  of  an  outsider  to  the  trainee  group. 

The  cause  of  the  training  weakness  must  be  identified.  Units  perform  poorly 
because  they  either  lack  the  knowledge  and/or  skills  (collective  and  individual)  to 
perform  the  tasks;  they  do  not  practice  the  tasks  and  associated  subtasks  (they  are 
“rusty”);  they  practice  the  tasks  under  unrealistic  conditions  and  without  specified 
standards  for  performance;  or  some  combination  of  all  three.  The  articulation  of  the  root 
cause  of  poor  performance  will  lead  the  participant  trainees  to  the  most  effective 
corrective  training  solution.  Hence,  performance  will  be  improved  in  an  efficient  maimer. 


Learning  Reinforcement 

The  third  element  of  an  effective  AAR  is  the  reinforcement  of  lessons  learned 
through  rehearsal  or,  if  at  all  possible,  re-training.  Research  has  shown  that  lessons 
reinforced  by  practice  are  understood  more  thoroughly,  retained  longer,  and  generalized 
to  other  situations  more  readily.  (Anderson,  1993,  and  Meister,  1976)  I  am  suggesting 
that  the  AAR  not  end  with  the  development  of  the  corrective  training  plan,  but  that  it 
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should  end  when  the  corrective  training  is  executed  to  the  performance  standards  outlined 
by  the  leader.  This  allows  the  greatest  opportunity  for  the  unit  to  learn.  Without  this 
reinforcement,  the  unit  does  not  maximize  its  performance  improvement  from  the  AAR. 

Undoubtedly,  the  execution  of  the  corrective  training  has  the  potential  to  extend 
the  training  exercise  a  great  deal  longer  than  originally  planned  if  there  are  many  and 
varied  performance  behaviors  to  correct.  Therefore,  the  corrective  training  tasks  to  be 
executed  should  be  limited  to  those  that  directly  relate  to  the  original  training  objectives. 
The  rest  should  be  plaimed  for  future  training. 

Another  consideration  for  training  plan  execution  is  the  fatigue  level  of  the  unit. 
Soldiers  that  have  expended  a  great  deal  of  energy  during  the  training  exercise  and  the 
AAR  may  not  have  much  left  for  an  intensive  corrective  training  exercise.  This 
highlights  the  need  for  the  appropriate  training  strategy  to  be  selected  during  discussion 
of  how  to  do  it  better.  Units  have  many  strategies  at  their  disposal.  They  range  from  a 
sand  table  exercise  with  only  leaders  to  a  fully  attended  MILES  situational  training 
exercise  against  an  OPFOR.  The  important  consideration  when  choosing  a  training 
strategy  is  to  maximize  the  resources  and  personnel  present  at  that  point  in  time.  It  can 
take  a  considerable  effort  to  re-coordinate  and  re-schedule  training  resources. 


Time 

Time  influences  the  effectiveness  of  AARs  in  three  ways:  the  timeliness  of  the 
AAR,  timeliness  of  the  retraining,  and  time  length  of  the  AAR.  Subject  matter 
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experience  and  studies  have  shown  that  the  effectiveness  of  the  feedback  is  inversely 
proportional  to  the  length  of  time  that  transpires  from  the  end  of  the  training  until  the 
feedback  is  received.  (Wortman,  C.  and  Loftus,  E.,  1981, 201  through  205)  The  same 
time  relationship  holds  true  for  retraining.  Feedback  has  the  most  impact  when  cause  and 
effect  relationships  and  training  conditions  are  still  fresh  in  the  trainees’  minds.  Hence, 
timeliness  of  the  AAR  is  important  to  accurately  diagnose  performance  and  the  timeliness 
of  the  retraining  is  important  to  practice  corrective  actions  with  respect  to  the  diagnosis. 

Logically,  time  has  a  third  impact  on  effectiveness.  As  Word  pointed  out,  the 
length  of  an  AAR  in  time  impacts  on  its  effectiveness.  The  attention  span  of  a  human  is 
limited,  especially  after  rigorous  training  in  a  fatigued  state.  Soldiers,  like  students  in  a 
classroom,  can  only  handle  an  average  of  three  to  four  learning  points  before 
attentiveness  dwindles.  This  means  that  there  is  only  time  for  three  to  four  problems  to 
be  identified,  discussed,  and  solutions  outlined  with  respect  to  future  training. 

A  time  efficient  method  of  identifying  a  problem  topic  for  discussion  is  to  use  the 
BDA  results.  The  cause  and  effect  relationships  that  produced  training  casualties  or 
equipment  losses  are  simple  and  efficacious  discussion  start  points.  Participants  are 
naturally  interested  in  discussing  how  and  why  each  became  or  caused  a  casualty. 
Additionally,  the  quest  to  minimize  casualties  during  an  action  is  an  obvious  and 
important  desire  of  the  unit.  Naturally,  everyone  wants  to  live.  The  accomplishment  of 
the  mission  with  the  fewest  losses  in  persormel  and  equipment  is  the  basic  purpose  of 
Army  training. 
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The  length  of  the  AAR  in  time  will  affect  unit  participation.  AAR  leaders  must 
guard  against  letting  discussions  drag  on.  If  participants  seem  disinterested  or  distracted, 
the  discussion  should  be  refocused,  redirected,  or  terminated. 


Foundations  of  Effectiveness  in  the  Theory  of  Learning 

The  theories  that  support  the  proposed  AAR  system  are  derived  from  educational 
research  and  cognitive  science.  The  intersection  of  these  two  separate  and  often  parallel 
disciplines  is  the  basis  for  the  AAR.  There  are  two  questions  that  educational  research 
revolves  around:  what  curriculum  content  to  teach  and  what  strategy  should  be  employed 
to  relay  this  content  to  the  student.  (Williams,  1996a)  Cognitive  science,  on  the  other 
hand,  offers  theories  that  explain  how  humeins  think  and  learn.  Thus,  the  combination  of 
these  theories  can  explain  the  effectiveness  and  efficiency  of  learning  for  a  specific 
teaching  strategy  and  curriculum  content.  In  other  words,  why  a  student  learns  more  at  a 
faster  pace  when  given  certain  information  in  a  specific  manner. 

Learning  effectiveness  and  efficiency  are  the  goals  of  any  instruction.  Given  that, 
it  is  beneficial  first  to  develop  an  understanding  of  these  two  terms.  Learning 
effectiveness  is  the  focus  of  knowledge  engineering  research  that  deals  with  how  well  a 
student  learns  the  curriculum  content.  This  is  determined  by  testing  the  student’s 
application  of  the  acquired  knowledge  in  some  maimer.  Whether  through  oral  and 
written  examination  or  physical  demonstration,  the  student  is  required  to  meet 
performance  standards  that  are  predetermined  by  the  instructor.  In  this  sense,  the  manner 
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in  which  the  student  leams  is  irrelevant  unless  it  prevents  him  from  mastering  all  of  the 
knowledge.  Thus,  effectiveness  relates  to  how  much  of  the  target  knowledge  units  are 
learned. 

Efficiency  in  learning  is  concerned  with  the  length  of  time  a  student  requires  to 
learn  a  specific  unit  of  knowledge.  Instructional  strategies  seek  to  maximize  the 
efficiency  of  learning  by  presenting  and  explaining  the  material  in  ways  that  the  student 
quickly  understands  it.  Most  of  the  instructional  research  is  on  how  to  increase  learning 
efficiency  because  knowledge  engineering  a  curriculum  is  extraordinarily  expensive  in 
effort  and  fimds.  Notwithstanding  the  time  and  effort  to  engineer  the  curriculxim,  a 
reorganization  of  domain  knowledge  units  requiring  corresponding  changes  in  teaching 
materials  and  instructor  training  (or  retraining)  is  also  monetarily  expensive. 

Cognitive  science  primarily  seeks  to  explain  how  hmnans  process  information  to 
accomplish  tasks,  solve  problems,  learn,  and  achieve  other  conscious  actions  or  thoughts. 
This  discipline  is  built  on  the  premise  that  the  human  brain  can  be  represented  as  an 
information-processing  system.  Basically,  this  system  explains  how  sensory  and  motor 
organs  receive  and  encode  symbols  from  the  environment  and  store  them  as  memory. 
These  memories  hold  interrelated  symbol  structures  that  are  processed  and  output  as  sets 
of  symbolic  structures  or  encoded  symbols  sent  back  to  the  sensory  and  motor  organs. 
(Williams,  1996a,  Langley  et  al.,  1987,  and  Card  et  al.,  1983)  In  this  manner,  the  system 
attempts  to  model  the  human  cognition  that  results  in  perception  and  sensory  inputs  to 
prompt  action. 
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The  idea  of  the  human  infonnation  processor  was  developed  by  Allen  Newell  and 
Herbert  Simon  at  Carnegie  Mellon  University  in  the  1960s  and  early  1970s.  It  gained 
popular  support  in  the  1970s  and  has  spawned  a  number  of  theories  and  models  since. 
One  of  the  most  successful  models  of  human  cognition  is  John  R.  Anderson’s  Adaptive 
Control  of  Thought  —  Rational  (ACT-R).  (Anderson,  1993)  This  model  explains  the 
imderlying  learning  processes  of  an  effective  AAR.  However,  a  number  of  cognitive 
science  terms  must  first  be  defined.  Afterward,  trainee  learning  is  explained  in  terms  of 
ACT-R.  ACT-R  is  explained  in  detail  in  Appendix  A,  Theoretical  Foimdations. 


Terms  of  Cognition 

Meta-Cognitive  Skill 

In  cognitive  science,  problem  solving  skill  is  also  known  as  meta-cognitive  skill. 
Specifically,  problem  solving  is  a  collection  of  cognitive  skills  used  to  achieve  the  two 
stages  of  the  problem  solving  process.  These  two  stages  are:  1)  problem  representation 
stage  and  2)  solution  stage.  (Simon  &  Newell,  1972)  Cognitive  research  provides  a 
convincing  amoimt  of  evidence  for  the  premise  that  the  human  problem  solving  process  is 
a  skill  that  can  be  learned  and  refined.  (Chapman  &  Allen,  1994;  Anderson,  1993; 

Newell,  1992;  Lesgold  et  al.,  1992;  Shute  and  Glaser,  1990;  and  Langley  et  al,  1987) 
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Problem  Space  and  Representation 


A  problem  is  represented  by  one  or  more  problem  spaces.  The  problem  space  is  a 
cognitive  term  used  to  explain  problem  solving.  Simon  and  Newell  (1972)  identify  a 
number  of  invariant  features  that  define  problem  spaces.  All  of  these  features  are  listed  in 
Appendix  A,  Theoretical  Foimdations.  The  important  feature  is  that  the  problem  space  is 
a  finite  set  of  knowledge  that  the  problem  solver  associates  with  or  recalls  given  a 
declarative  fact.  It  is  what  the  person  knows  and  can  consider.  Therefore,  the  space  is 
constantly  changing  and  can  be  represented  as  a  series  of  knowledge  states  where  each 
state  is  generated  from  a  finite  (usually  small)  set  of  objects,  properties,  and 
environmental  forces,  and  their  interrelationships. 

The  knowledge  states  that  make  up  the  problem  space  are  referred  to  as  problem 
states.  These  problem  states  are  initial,  intermediate,  and  final  situation  problem 
representations.  Thus,  the  problem  space  consists  of  an  initial  (understanding) 
representation  of  the  problem,  the  final  endstate  (solution),  and  any  number  of  iterative 
step  states  that  connect  the  two  (initial  understanding  ->  final  solution).  The  problem 
state  that  represents  the  final  solution  is  called  the  goal  or  goal  state.  (Langley  et  al., 

1987, 9;  Newell,  1992;  Anderson,  1983;  Card  et  al.,  1983;  and  Simon  &  Newell,  1972, 
811  and  812) 

Each  problem  state  employs  one  or  more  representation  techniques  to  define 
itself.  Langley  et  al.  (1987)  define  the  representation  within  the  problem  state  as  “a 
scheme  for  holding  information  in  a  memory,  combined  with  processes  for  operating  on 
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that  information  to  access  it,  alter  it,  or  draw  inferences  from  it.”  (Langley  et  al.,  1987, 
315)  The  primary  representation  schemes  recognized  are  list  structures,  sentential 
structures,  and  pictorial  or  imaginal  structures.  (Langley  et  al.,  1987,  315  through  335) 
The  bottom  line  is  that  humans  use  different  representation  schemes  to  represent  a 
problem  state  within  a  given  problem  space.  For  a  complete  discussion  of  and  evidence 
for  problem  spaces  and  representations  see  Langley  et  al.  (1987)  and  Williams  & 
Reynolds  (1991). 

Goals 

The  goal  state  of  a  problem  space  is  always  defined  with  respect  to  that  problem 
space.  In  the  same  sense,  a  problem  is  always  defined  with  respect  to  the  environment  it 
occurs,  or  is  realized,  in.  Thus,  the  problem  space  provides  the  context  for  the  goal  state. 
Without  this  context,  the  goal  state  would  have  no  meaning.  The  reason  for  this 
seemingly  obvious  statement  is  that,  like  many  other  tasks  and  experiences  in  life,  where 
one  ends  up  or  finishes  depends  largely  upon  where  or  how  the  goal  was  defined.  The 
presence/absence  of  a  goal  with  respect  to  the  problem  state  (or  a  task  to  be  learned) 
determines  the  effectiveness  and  efficiency  of  problem  solution  (and  learning  of  the  task). 
(Newell,  1992;  Williams  &  Reynolds,  1991;  Anderson,  1986;  and  Jeffries  et  al.,  1981) 

In  the  above  terms,  learning  can  be  defined  as  a  problem  solving  process. 
Learning,  as  used  in  this  thesis,  is  the  acquisition  of  a  new  skill  or  knowledge  that  can 
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then  be  applied  in  a  wide  range  of  conditions  and  environments.  Implied  here  is  that  the 
knowledge  and  skill  of  previous  experience  is  retrieved  for  use  in  the  acquisition  process. 
Essentially,  this  process  is  problem  solving.  The  initial  state  is  defined  by  previous 
experience  (what  is  already  known).  The  goal  state  is  the  target  skill  or  knowledge  to  be 
acquired.  The  intermediate  states,  between  the  initial  and  goal  state,  are  the  iterations  of 
step  states  to  get  from  the  initial  state  to  the  goal. 

What  may  not  be  apparent  from  the  above  definitions  is  that  problem  states  can 
subdivide  into  multiple  sub-problem  states.  Just  as  people  consciously  break  problems 
into  manageable  components,  so  does  the  cognitive  control  decompose  problem  states. 
These  definitions  and  their  interactions  basically  subscribe  to  Newell’s  theory  of  problem 
solving  called  SOAR.  (Newell,  1992)  SOAR  is  a  problem  solving  theory  of  cognition 
that  explains  acquisition  of  text  editing  skills  that  are  represented  with  the  GOMS  (Goal- 
Operator-Method-Selection  rule)  knowledge  representation  technique.  (Card  et  ah,  1983, 
and  Williams  &  Kotnour,  1993) 

Compelling  evidence  for  the  importance  of  problem  spaces  in  problem  solving  is 
presented  in  Langley,  Simon,  Bradshaw,  and  Zytkow  (1987),  Scientific  Discovery: 
Computational  Explorations  of  the  Creative  Processes.  In  their  research,  Langley  et  al. 
modeled  a  number  of  significant  scientific  discoveries  with  computer-based  production 
systems.  The  production  rules  were  based  on  the  scientific  method  -  the  systematic 
assessment  of  a  problem,  followed  by  hypothesis  formulation,  data  collection,  and 
hypothesis  testing.  Langley  et  al.’s  research  centered  on  the  hypothesis  that  “scientific 
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discovery  can  be  explained  as  a  form  of  problem  solving;  that  its  basic  processes  are 
‘normal’  problem-solving  processes,  adapted  to  the  particular  characteristics  of  the 
domain  of  discovery.”  (Langley  et  al.,  1987, 338)  The  following  is  a  list  of  discoveries 
that  their  artificial  intelligence  (Al)-based  scientists  rediscovered:  Planck’s  discovery  of 
the  law  of  blackbody  radiation  and  its  consequences  with  respect  to  quantum  action; 
Newton’s  discovery  of  the  law  of  universal  gravitation;  Kepler’s  third  law  of  planetary 
motion;  Boyle’s  Law;  Galileo’s  law  of  uniform  acceleration;  Ohm’s  Law;  Snell’s  law  of 
refraction;  Black’s  specific-heat  law,  and  conservation  of  momentum;  Joule’s  law  of 
energy  conservation;  Dalton’s  atomic  theory;  and  the  qualitative  laws  of  acids  and  bases 
(that  preceded  fiie  and  prompted  the  later  discovery  of  their  empirical  relationships).  This 
collection  of  discoveries  can  be  categorized  as  finding  quantitative  laws,  generating 
qualitative  laws,  inferring  the  components  of  substances,  and  formulating  structural 
models. 

Besides  proving  that  the  general  approach  of  scientific  inquiry  applies  across  a 
broad  spectrum  of  disciplines,  Langley  et  al.  found  something  more  profoimd.  Each  of 
these  discoveries  was  based  not  on  any  particular  genius  or  magical  “flash  of  insight,”  but 
on  how  one  defined  the  problem  space.  (Langley  et  al.,  1987, 312, 328,  and  339) 
Specifically,  they  demonstrated  that  the  generation  of  research  problems  is  a  problem¬ 
solving  process  that  can  be  formalized  and  approached  in  the  same  manner  as  other 
problem-solving  processes.  Hence,  the  discovery  of  the  correct  problem  space  is  the  real 
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accomplishment  and  it  is  this  problem  definition  that  is  the  most  important  factor  in  a 
learning  experience. 

As  a  leader  in  cognitive  science,  John  Anderson’s  work  is  important  in  this 
context  because  he  conclusively  showed  that  cognitive  skills  are  realized  by  production 
rules.  Thus,  the  foundation  of  Langley  et  al.’s  AI  programs  is  validated  by  Anderson 
(1993).  Anderson’s  ACT-R  is  the  most  successful  model  at  describing  human  knowledge 
acquisition,  performance,  and  transfer  to  date. 


Learning  and  ACT-R  Theory 

In  Intelligent  Simulation  Training  Systems  Design.  Kent  E.  Williams  (1996a) 
reviews  current  cognitive  theories  of  knowledge  representation,  knowledge  acquisition, 
and  problem  solving.  Theories  of  cognition  attempt  to  explain  how  humans  process  and 
recall  information  to  acquire  skill.  Problem  solving,  or  meta-cognitive  skill,  is  how 
humans  use  the  information  processed  or  recalled.  These  cognitive  theories  describe  the 
underlying  mechanism  of  the  meta-cognitive  processes  associated  with  learning. 

Again,  ACT-R  is  a  very  successful  model  at  describing  knowledge  acquisition, 
performance,  and  transfer  in  a  knowledge-based  (task  based)  environment.  (Williams, 
1996a)  Anderson  explains  ACT-R  and  outlines  its  success  in  Rules  of  the  Mind.  (1993). 
He  posits  that  the  success  of  modeling  human  learning  and  performance  with  ACT-R 
validates  its  production  system  structure.  With  this  conclusion,  ACT-R  is  offered  as 
proof  that  human  cognitive  skills  are  realized  by  production  rules.  (Anderson,  1993)  An 
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earlier  version  of  ACT-R,  ACT*,  was  independently  validated  and  verified  as  an 
individual  basis  for  learning  by  Bonarini  &  Filippi  (1993)  and  in  a  classroom  setting  with 
strong  learning  effects  by  Schofield  et  al.  (1990).  Another  conclusive  and  comprehensive 
experiment  that  verifies  and  validates  Anderson’s  theory  with  respect  to  problem  solving 
is  Swan  &  Black  (1993).  This  research  tested  meta-cognitive  skill  improvement  as  a 
result  of  knowledge-based  declarative  instruction/tutoring  and  procedural  practice. 

Production  rules  (or  productions)  are  condition-action  pairs  or  if-then  rules. 
(Anderson,  1993, 4  and  5)  The  condition  side  specifies  the  circumstances  under  which 
the  action  side  will  apply.  The  action  side  specifies  what  to  do  if  the  circumstances  are 
satisfied.  These  productions  form  the  basic  imits  of  skill.  In  other  words,  productions  are 
what  a  student  learns.  Furthermore,  the  analysis  to  identify  the  production  rules  in  a 
knowledge  domain  and  then  the  teaching  of  those  rules  leads  to  efficient  learning. 
(Anderson,  1993;  Williams,  1996a;  Williams  &  Reynolds,  1991;  and  Kieras  &  Bovair, 
1986)  The  identification  of  critical  knowledge  units  is  the  purpose  of  knowledge 
engineering. 

More  specifically,  ACT-R  is  a  computational  model  of  productions  with  the 
premise  that  learning  is  based  upon  what  is  already  known  -  prior  experience.  This 
experience  is  stored  in  procedural  memory  and  consists  of  the  production  rules  with 
similar  goals  and/or  conditions.  (Anderson,  1993)  With  more  elaboration  and  practice, 
these  productions  become  situation  specific  allowing  the  experience  to  be  situation-based. 
The  subject  learns  by  iteratively  applying  that  experience  to  a  given  problem. 
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ACT-R’s  implications  for  skill  acquisition  are  summarized  below.  These  derive 
directly  from  the  argument  that  human  cognitive  skills  are  realized  by  production  rules. 

•  The  knowledge  underlying  a  skill  begins  in  an  initial  declarative 

form  (an  elaborated  example),  which  must  be  interpreted  (problem 
solved  by  analogy)  to  produce  performance. 

•  As  a  function  of  its  interpretive  execution,  this  skill  becomes 

compiled  into  production-rule  form. 

•  With  practice,  individual  production  rules  acquire  strength  and 

become  more  attuned  to  the  circumstances  in  which  they  apply; 
they  become  situation  based. 

•  Learning  complex  skills  can  be  decomposed  into  the  learning 

functions  associated  with  individual  production  rules.  (Anderson 
etal.,  1993,  143) 

Given  that  production  units  are  the  basic  units  of  skill,  performing  a  detailed 
cognitive  task  analysis  with  respect  to  productions  can  improve  performance.  More 
efficient  learning  is  achieved  by  using  this  analysis  (of  the  production  units)  to 
specifically  incorporate  the  identified  knowledge  chunks  into  the  curriculum.  (Anderson, 
1993;  Williams,  1996a;  and  Williams  &  Reynolds,  1991)  Thus,  the  question  arises  of 
why  task  analyses,  in  production  level  detail,  have  not  been  employed  in  training 
systems?  The  short  answer  to  this  question  is  that  the  task  analysis  process  is  an 
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extremely  expensive  venture  in  both  time  and  resources.  (Williams,  1996a,  Williams, 
1995,  and  Chapman  &  Allen,  1994) 

Cognitive  and  meta-cognitive  research  strongly  supports  the  focusing  of  an  AAR 
discussion  on  a  performance  problem-solution  to  learn.  (Roschelle  &  Teasley,  1995; 
Anderson,  1993;  Swan  &  Black,  1993;  Newell,  1992;  Williams  &  Reynolds,  1991;  and 
Langley  et  al.,  1987)  The  goal  of  the  discussion  is  to  develop  a  solution  of  corrective 
actions.  It  is  a  problem  solving  discussion. 

The  “what  happened”  and  “why  it  happened”  specified  in  doctrinal  guidance  are 
only  important  in  terms  of  diagnosing  performance  to  identify  the  needed  corrective 
behaviors.  The  learning  in  terms  of  improved  performance  derives  directly  from 
strengthening  the  productions  that  define  the  corrective  action.  It  is  the  corrective  action 
that  produces  the  desired  effect  of  improving  performance.  (Williams  &  Reynolds,  1991, 
and  Black  et  al.,  1987) 

The  knowledge  that  underlies  a  skill  begins  as  an  elaborated  example.  (Anderson, 
1993)  Verbal  elaboration  allows  the  new  knowledge  to  be  associated  with  past 
experience  in  the  proper  context.  In  terms  of  military  performance,  this  elaboration 
allows  an  individual  to  associate  a  corrective  action  (behavior)  with  a  proper 
interpretation  of  perceptual  cues.  In  other  words,  if  (cues)  a  squad  occupies  a  support  by 
fire  position  and  cannot  suppress  all  of  the  enemy  positions,  then  (corrective  action)  the 
squad  must  reposition  to  obtain  observation  of  all  enemy  positions.  Here  the  members  of 
the  squad  must  be  able  to  articulate  what  perceptual  cues  will  indicate  that  they  cannot 
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suppress  all  of  the  enemy  positions.  In  this  example,  the  first  indieators  will  most  likely 
be  one  or  both  of  the  squad  receiving  effective  fire/casualties  and/or  a  message  that  their 
fires  are  ineffective.  The  point  is  if  they  can  articulate  the  conditions  and  actions,  then 
they  will  be  able  to  perform  the  corrective  action  upon  recognizing  the  situation.  The 
trainees’  ability  to  proceduralize  the  learning  point  (conditions-corrective  action)  is  a 
function  of  their  interpretive  execution  of  that  specific  task. 

The  next  step  is  to  physically  practice  the  corrective  action  before  the  memory  of 
the  declarative  elaboration  decays.  As  Anderson  (1993)  shows,  sooner  is  better.  He 
empirically  demonstrated  that  a  knowledge  unit’s  retention  and  recall  for  use  are  a 
function  of  frequency  of  use  and  time  since  last  used.  Moreover,  retention  and  recall  can 
be  improved  as  a  power  function  of  trials  (number  of  times  of  repeated  use),  but  will 
decay  as  a  function  of  delay.  Thus,  efficient  learning  calls  for  the  significant  behaviors  to 
be  identified  and  reinforced  immediately  after  the  training  experience.  If  a  specific 
performance  outcome  is  desired,  then  the  modification  of  behaviors  to  achieve  it  must 
also  be  immediate.  Practice  until  one  gets  it  right. 

The  timely  execution  of  corrective  actions  to  modify  behavior  is  critical  to 
learning.  A  unit  must  elaborate  amongst  themselves  what  tasks  will  be  performed,  given 
a  specific  set  of  conditions,  to  achieve  the  designated  goal.  The  elaboration  allows 
individuals  to  associate  (understand)  the  corrective  actions  with  past  experience  and 
acquire  the  same  behaviors  to  achieve  the  goal.  The  next  step  is  physical  practice  to 
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reinforce  the  newly  formed  productions  of  common  understanding.  Hence,  learning 
reinforcement  amounts  to  repetitive,  mental  and  physical  practice. 

Anderson  made  one  other  significant  finding  relevant  to  AARs  and  learning;  the 
learning  process  was  made  more  efficient  by  instructor  guidance.  (Anderson,  1993, 159) 
Students  without  the  benefit  of  a  tutor  took  the  longest  to  learn.  Those  that  did  learn  “on 
their  own”  did  not  know  the  subject  any  better  than  those  who  received  direction  and 
guidance,  they  simply  took  longer.  Thus,  there  is  no  benefit  gained  from  learning  a 
number  of  wrong  ways  before  discovering  a  right  way.  This  is  counterintuitive  to  most 
teachers/people  who  believe  that  there  is  some  benefit  derived  from  knowing  why  things 
are  wrong  or  what  will  not  work  when  mastering  a  subject  domain.  Anderson’s  finding 
does  not  totally  reject  this  paradigm.  He  found  that  knowing  the  correct  answer  first  led 
to  a  quicker  discovery  and  understanding  of  the  relationships  between  correct  and 
incorrect  solutions.  Again,  the  strongest  independent  confirmation  of  Anderson’s  finding 
is  Swan  &  Black  (1993). 

The  finding  that  guided  discovery  is  more  efficient  justifies  the  AAR  leader’s 
role.  This  justification  assumes  that  the  AAR  leader  possesses  the  knowledge 
(experience)  to  guide  the  trainee  discussion  to  a  correct  solution  of  performance 
problems.  Here  lies  an  inherent  problem  of  a  novice  leading  an  AAR.  If  the  AAR  leader 
is  not  familiar  enough  with  doctrine  to  apply  it  to  a  tactical  performance  problem, 
learning  efficiency  is  decreased  as  he  and  the  trainees  stumble  through  a  number  of 
incorrect  solutions.  In  this  case,  the  AAR  participants’  situation  is  similar  to  that  of  a 
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team  of  scientists  endeavoring  to  discover  the  relationships  that  will  describe  a  particular 
phenomenon.  The  situation  calls  for  systematic,  goal  oriented  search  of  the  problem 
space. 


Conclusion 

The  AAR  is  a  critical  part  of  the  Army  training  system.  It  evolved  as  the  Army’s 
ability  to  simulate  combat  conditions  in  training  improved.  The  AAR  was  fully  defined 
and  disseminated  as  doctrine  with  the  establishment  of  the  CTCs  —  NTC,  JRTC,  and 
CMTC.  The  CTCs  have  set  the  example  in  how  to  train  and  implement  the  AAR  as  a 
vehicle  for  performance  feedback.  The  exact  form  of  this  example  is  explored  in  Chapter 
III. 

The  effectiveness  of  the  AAR  is  derived  from  four  elements:  discussion 
participation,  discussion  focus,  timeliness,  and  learning  reinforcement.  The 
interdependence  of  these  elements  are  just  as  important  as  an  element  is  by  itself.  The 
first  is  the  imit’s  participation  in  a  discussion  to  discover  and  define  performance 
problems,  develop  solutions,  and  formulate  corrective  action  plans.  This  is  critical 
because  it  allows  the  unit,  not  the  AAR  leader  or  outside  observer,  to  articulate  what 
happened,  why,  and  how  to  fix  it.  Hence,  the  unit  realizes  the  problem,  takes  ownership 
of  the  solution,  and  learns  in  the  process.  Learning  is  facilitated  by  discussion 
participation  which  allows  unit  members  to  explain  the  relationships  that  affect 
performance  to  one  another  in  their  own  terms,  at  their  own  level  of  understanding. 


67 


Second,  the  AAR  should  be  a  professional  and  dynamic  discussion  focused  on 
solving  performance  problems.  The  trainees  learn  by  verbally  elaborating  the  problem 
solution  in  terms  of  their  past  experience.  Specifically,  the  focus  of  the  discussion  is  on 
the  elaboration  of  the  relationship  between  past  experience  and  the  solution.  Focusing  on 
one  problem  at  a  time,  the  end  state  of  the  AAR  should  be  that  the  unit  has  collaborated 
to  achieve  a  clear  understanding  of: 

1 .  JVhat  happened  during  the  training  exercise; 

2.  The  conditions  and  behaviors  that  caused  measured  outcomes  (results); 

3.  How  the  behaviors  or  conditions  could  be  modified  to  achieve  the  desired 

performance  outcomes;  and 

4.  A  training  plan  to  sustain  successful  performance  and  correct  deficient 

performance. 

The  training  plan  should  include  the  tasks  to  be  trained,  constraining  and  limiting 
conditions  to  be  trained  under,  and  standards  of  performance  tied  to  the  purpose  and 
objectives  of  the  training.  Furthermore,  the  unit  should  perceive  this  plan  as  its  own 
creation  in  accordance  with  its  needs  and  not  dictated  by  superior  officers,  the  AAR 
leader,  or  a  computer  based  system. 

Third,  the  AAR  must  be  conducted  within  a  short  time  period  of  the  training  event 
so  that  the  learning  will  have  significance.  This  will  help  the  unit  to  efficiently  reinforce 
successful  performance  and  correct  weak  performance.  Additionally,  the  AAR  must  be 
conducted  within  a  unit’s  fatigue  capacity.  It  must  be  of  a  reasonable  length  of  time  that 
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does  not  exceed  the  unit’s  capability  to  concentrate  on  problem  solving.  Time  also 
affects  the  fourth  element  of  effectiveness,  learning  reinforcement. 

The  AAR  must  allow  for  the  timely  practice  of  specified  solutions  at  the  earliest 
possible  opportunity.  Most  importantly,  the  solutions  are  associated  with  past  experience 
by  mental  rehearsal  as  well  as  physical  performance.  The  sooner  that  the  elaborated 
actions  can  be  physically  performed  (practiced),  the  stronger  the  solution  representations 
will  be  in  the  trainees’  minds.  The  timely  execution  of  this  corrective  training  is  critical 
to  the  reinforcement  of  learning.  If  a  unit  does  not  perform  the  corrective  or  reinforcing 
training  as  it  plans,  then  the  learning  points  and  previous  training  are  relegated  to 
unimportance  and  have  been  a  waste  of  time. 

Chapter  III  examines  the  CTC  AAR  system  and  its  implementation  with  respect 
to  effectiveness  and  simulation  training  system  support  components. 
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CHAPTER  III 


CTC  AFTER  ACTION  REVIEWS 


"The  truth  is  sought,  regardless  of  whether  pleasant  or  unpleasant." 

—  LTG  Leslie  J.  McNair 

AARs  are  the  US  Army’s  mechanism  for  individual  and  unit  performance 
feedback  in  training  and  combat.  (DA,  FM  25-101, 1990,  and  TC  25-20,  1993)  As  such, 
the  CTCs  have  set  the  example  in  the  conduct  of  AARs  and  realistic  training  for  the  rest 
of  the  Army  and  the  training  system  research  and  development  community.  For  this 
research,  the  scope  of  the  Army’s  training  domain  includes  combat  and  combat  support 
units  who  conduct  training  in  live,  virtual,  and  constructive  simulation  environments. 
These  units  are  not  restricted  to  one  training  environment  but  may  also  train  in  a 
combination  of  environments.  Whatever  the  setting,  leaders  design  and  execute  battle 
focused  training  as  realistically  as  possible  so  that  performance  feedback  is  meaningful. 
The  AAR  is  the  vehicle  for  this  feedback  in  all  types  of  training.  (DA,  FM  25-101, 1990) 
In  this  respect,  it  is  beneficial  to  review  what  the  AAR  components  are  at  the  CTCs  and 
how  this  example  is  applied  to  small  unit  training  simulation  systems. 
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Currently,  AARs  come  in  a  number  of  shapes  and  sizes  for  training  simulation 
systems.  The  Simulation,  Training,  and  Instrumentation  Command  (STRICOM)  and 
commercial  contractors  are  developing  a  number  of  simulation  training  systems  to  meet 
training  needs.  To  a  varying  degree,  each  has  its  own  version  of  an  AAR  “system.” 

Small  unit  simulation  training  systems  are  no  different.  The  Simulation  Network 
(SIMNET)  was  purposefully  built  without  an  AAR  system  because  the  requirements  had 
never  been  defined.  (Meliza,  L.,  1996,  3)  The  Close  Combat  Tactical  Trainer  (CCTT) 
will  replace  SIMNET  as  a  virtual  simulation  trainer  for  the  platoon  with  the  additional 
capability  to  train  company  teams.  Its  AAR  system  requirements  are  still  being 
developed.  (SHERIKON,  February  1996,  and  STRICOM,  July  1996) 

Since  CTC  AARs  serve  as  the  example  for  Army  combat  training,  the  purpose  of 
this  chapter  is  to  demonstrate  what  they  consist  of  in  light  of  the  elements  of  effectiveness 
in  the  previous  chapter.  Once  the  example  is  clear,  current  small  unit  simulation  and 
training  systems  will  be  surveyed  to  identify  their  components. 

This  chapter  and  its  conclusions  are  based  upon  two  assumptions.  The  first  is  that 
US  Army  combat  units  train  lAW  doctrine  and  seek  to  conduct  AARs  lAW  this  doctrine. 
Given  that  officers  and  noncommissioned  officers  receive  very  little  formal  training  on 
how  to  conduct  an  AAR  (personal  experience),  each  learns  by  the  example  of  others.  It 
follows  then,  that  the  principal  AAR  example  is  established  at  the  CTCs.  A  number  of 
active  duty  US  Army  officers  confirmed  that  most  division  level  AAR  guidance  is  a 
paraphrase  of  the  CTC  AAR  guidance.  (Personal  communication  with:  Bedell,  B.,  30 
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October  1996,  Lartigue,  L.,  2  November  1996,  Lipinski,  M.,  2  November  1996,  Lusher, 
R.,  30  October  1996,  and  McCarthy,  J.,  30  October  1996)  It  also  follows  that  the  CTC 
AAR  models  define  the  Army  training  commimity’s  expectations  of  what  an  AAR  should 
be. 

The  second  assumption  is  that  AAR  systems  built  for  simulation  training  systems 
are  designed  to  meet  the  requirements  of  Army  doctrine  and  the  training  user’s  needs. 
Combined  with  the  first  assumption,  this  implies  that  since  the  user’s  needs  and 
expectations  are  defined  by  the  CTC  AAR  example,  that  example  also  serves  as  a  model 
for  simulation  training  system  AARs.  This  is  supported  by  the  fact  that  STRICOM  and 
AAR  system  contractors  constantly  reference  the  CTC  AARs  in  written  and  verbal 
presentations.  (US  ARI,  1997,  STRICOM,  1996f,  BCTD,  1996b,  STRICOM  et  al.,  1996, 
and  STRICOM,  1994) 


CTC  AARs 

Both  the  JRTC  and  the  NTC  produee  AAR  guidance  on  videotape  for  the  OCs 
who  are  designated  to  lead  the  AAR.  Each  tape  is  intended  for  use  with  FM  25-101  and 
TC  25-20  and  provides  the  AAR  leader  with  the  techniques  and  proeedures  for  preparing 
and  conducting  an  AAR.  (JRTC/BDM,  1993,  NTC,  1994,  and  NTC/BDM,  1992)  AARs 
at  both  CTCs  are  led  by  a  senior  OC  who  has  a  certain  amount  of  latitude  in  interpretation 
of  doetrine  and  how  guidance  is  implemented.  However,  JRTC  and  NTC  differ  on  how 
they  present  the  material  and  the  specific  points  emphasized  in  the  AAR. 
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JRTC 


The  JRTC  AAR  guidance  states  that  the  “key  to  an  effective  AAR  is  the  OC,”  the 
AAR  leader.  (JRTC/BDM,  1993)  Although  the  guidance  does  not  specifically  define 
what  makes  an  AAR  effective,  it  implies  that  effectiveness  can  be  achieved  by  the 
following  guidance  of  what  an  AAR  is: 

•  A  professional  discussion  of  an  event  or  task  that  focuses  on 

performance  standards. 

•  Allows  soldiers  to  evaluate  for  themselves  what  happened  and  why. 

•  Concentrates  on  how  to  sustain  strengths  and  improve  on  weaknesses. 

•  Is  a  tool  leaders  can  use  to  extract  the  maximum  benefit  from 

experiences  of  training. 

•  It  is  important  to  include  all  members  of  the  unit  in  the  AAR. 

(JRTC/BDM,  1993) 

These  support  the  elements  of  effectiveness  defined  at  the  end  of  the  second  chapter  - 
participation  and  the  discussion  to  clearly  define  weaknesses/strengths  and 
corrective/sustainment  action. 

The  preparation  guidance  focuses  on  the  OC  identifying  two  to  four  key  issues  to 
be  covered  in  the  AAR.  These  issues  should  be  related  to  the  critical  actions  and  their 
effects  that  occurred  during  training.  The  guidance  states  that  the  discussion  of  how  to 
fix  these  issues  is  the  “heart  of  the  AAR.”  To  achieve  this,  the  AAR  time  is  broken  down 
as  follows:  25%  of  the  time  spent  on  what  happened,  25%  spent  on  why  it  happened,  and 
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50%  spent  on  how  to  do  it  better.  (JRTC/BDM,  1993)  The  JRTC  suggests  that  a  specifie 
AAR  format  (Table  3.1,  JRTC  AAR  Format)  be  used  to  aehieve  this  guidance. 


Table  3.1,  JRTC  AAR  Format 


1 .  Introduction.  Specify  purpose  and  length  of  the  AAR.  Introduce  the  key  issues. 

2.  Review  the  mission  or  task  performed  and  the  associated  doctrinal  training 

standards. 

3 .  The  unit  commander  explains  his  mission  and  plan. 

4.  The  OPFOR  representative  explains  his  mission,  plan,  and  execution  of  the  plan. 

The  OPFOR  representative  then  presents  observations  of  the  unit’s 
performance. 

5.  What  happened?  The  OC  summarizes  the  key  events  and  their  results  in  terms 

ofBDA. 

6.  Discussion  of  2-4  key  issues. 

a)  Select  the  key  issues  that  affected  the  battle  outcome  the  most. 

b)  Emphasize  why  it  happened  and  how  to  do  it  better. 

c)  Link  weaknesses  to  the  plan,  preparation,  and  execution  phases  of  the 

training  for  easy  transition  into  how  to  do  it  better. 

d)  The  OC  can  organize  the  discussion  by  battlefield  operating  system  (BOS). 

7.  Senior  noncommissioned  officer  OC  presents  observations  of  individual  task 

performance. 

8.  Key  leader  self-assessment:  allow  unit  to  input  what  they  will  fix  and  sustain; 

record  and  ensure  individuals  sign  up  to  fix/sustain  items  or  behaviors  that 
they  control;  force  the  unit  to  explain  how  they  plan  to  fix  the  shortcomings  or 
weaknesses;  and  finish  with  a  clear  bottom  line. 

9.  Summarize  training  standards  and  key  lessons  learned. 

10.  Cover  safety  issues  and  any  fratricide. 

________ 
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JRTC  guidance  places  the  onus  of  selecting  the  discussion  issues  on  the  OC.  The 
OC  selects  the  event  results  or  performance  outcomes  that  correspond  to  a  training 
standard,  specified  in  the  doctrinal  manuals,  as  the  issues.  This  allows  the  discussion  of 
strengths  and  weaknesses  to  be  founded  on  objective  results  that  are  not  questionable.  It 
also  is  an  efficient  method  to  identify  discussion  topics  when  the  trainees  are  hesitant  to 
do  so. 

The  trainees  receive  performance  feedback  from  three  different  people.  The  AAR 
leader  OC,  the  OPFOR  representative,  and  the  senior  noncommissioned  officer  OC.  This 
feedback  is  presented  via  terrain  models,  large  maps  with  operational  graphics,  overhead 
projectors,  flip  charts,  video  and  audio  recordings,  and  computers.  The  guidance 
specifically  states  that  the  use  of  these  training/visual  aids  “must  facilitate 
discussion. .  .and  should  be  limited  to  those  that  will  assist  in  making  the  point  or 
clarifying  the  situation.  Too  many  training  aids  will  detract  from  the  AAR  and  confuse 
participants.”  (JRTC/BDM,  1993) 

Although  the  OPFOR  representative  is  initially  involved,  the  free-play  training 
exercise  considerations  prevent  him  from  staying  for  the  entire  AAR.  However,  frie 
trainees  are  allowed  to  question  and  discuss  issues  with  him  before  he  leaves. 
(JRTC/BDM,  1993)  It  is  important  that  the  OC  brief  the  OPFOR  representative  on  the 
selected  key  issues  so  that  he  can  present  the  corresponding  enemy  actions.  The  OPFOR 
presentation  can  also  lead  to  discussion  of  key  issues  that  the  participants  identify.  If  this 
happens,  the  OC  or  the  unit  must  prioritize  the  issues  to  discuss  as  there  will  not  be 
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enough  time  to  cover  them  all.  The  guidance  for  AAR  length  is  one  hour  for  platoon  and 
two  hours  for  company  level  AARs.  (JRTC/BDM,  1993)  A  mission  of  two  to  three  days 
is  the  normal  period  of  training  that  is  covered  by  the  AAR  at  JRTC. 

The  OC  facilitates  the  key  leader  self-assessment  by  prompting  individuals  to 
assume  responsibility  for  implementing  corrective  behavior  for  weaknesses  or  reinforcing 
behavior  for  strengths.  It  is  important  that  the  OC  ensures  the  soldiers  only  assume 
responsibility  for  those  issues  that  they  can  affect.  (JRTC/BDM,  1993)  In  this  manner, 
responsibility  for  both  short  term  and  long  term  solutions  is  fixed. 

The  AAR  is  concluded  with  safety  and  fratricide  issues  that  occurred  during  the 
training  exercise.  (JRTC/BDM,  1993)  This  is  a  reinforcement  of  the  Army’s  policy  to 
train  safely  and  keep  safety  consideration  at  the  forefront  of  leaders’  minds.  (DA,  FM  25- 
100,  1988b,  1-3  and  4-3) 


NTC 

To  focus  the  OC,  the  NTC  AAR  guidance  defines  an  AAR  as  “a  structured 
review  process  that  allows  participants  to  discover  for  themselves  what  happened,  why  it 
happened,  and  how  it  can  be  done  better.”  (NTC/BDM,  1992)  The  NTC  guidance  states 
that  there  are  five  standards  that  define  a  successful  AAR. 

1 .  Allow  the  unit  to  self  discover. 

2.  Establish  cause  and  effect  relationships. 
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3.  Facilitate  problem  solving. 

4.  Base  the  AAR  on  doctrine,  tactics,  techniques,  and  procedures. 

5.  Always  foster  continuous  improvement  by  the  unit.  (NTC,  1994,  and 
NTC/BDM,  1992) 

To  achieve  these  standards,  the  OC  may  format  the  AAR  to  cover  key  issues,  the 
Battlefield  Operating  System  (BOS),  a  chronological  execution  of  events,  or  a 
teaching/lecture  of  learning  points.  The  OC  chooses  the  format  that  best  fits  the  AAR 
issues  and  causes  the  unit  to  solve  the  problem.  The  guidance  recommends  either 
selecting  key  issues  or  using  file  BOS  to  identify  strengths  and  weaknesses.  It 
recommends  against  using  the  chronological  review  of  events  since  issues  are  not 
highlighted  and  this  method  uses  a  lot  of  time.  The  teaching/lecture  method  is  to  be  used 
only  if  the  unit  is  significantly  below  Army  performance  standards  and  is  ineapable  of 
self  criticism.  (NTC/BDM,  1992) 

The  NTC  guidance  also  specifies  that  the  plan/prepare/execute  strueture  be 
applied  to  the  AAR  discussion.  This  structure  allows  units  to  categorize  the  discussion  so 
that  performance  tasks  can  be  identified  as  the  cause  of  specified  effects.  (NTC,  1994, 
and  NTC/BDM,  1992)  Hence,  the  question  of  why  an  event  occurred  ean  be  easily  tied 
to  the  good/poor  performance  of  an  individual  or  collective  training  task.  This  also 
allows  the  unit  to  easily  prescribe  training  to  correct  or  reinforce  performance.  The 
guidance  emphasizes  that  the  AAR  is  not  a  critique  and  should  focus  on  meeting 
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doctrinal  performance  standards.  (NTC,  1994,  andNTC/BDM,  1992)  Table  3.2  lists 
other  “do’s  and  don’ts”  for  the  NTC  OC. 


Table  3.2,  NTC  AAR  Do’s  and  Don’ts 


Do 

Don’t 

Establish  an  agenda 

Talk  down  to  the  unit 

Encourage  note  taking 

Argue  with  the  unit 

Record  key  issues/solutions 

Do  all  the  talking 

Be  flexible 

Allow  MILES  sniveling 

Summarize  the  AAR 

Use  subjective  assessments  of  unit 
actions 

Present  a  professional  appearance 

(NTC/BDM,  1992) 

The  NTC  guidance  emphasizes  the  use  of  a  terrain  model  as  critical  to  the  AAR. 
The  terrain  model  combined  with  concept  sketches  of  events  or  actions  enables  the 
trainees  to  clearly  visualize  what  happened  and  why.  (NTC/BDM,  1992)  The  trainees  are 
also  better  able  to  collectively  identify  cause  and  effect  relationships  with  a  common 
vision  of  the  battlefield.  The  other  aids  that  are  emphasized  for  platoon  and  company 
level  AARs  are  photographs  and  large  scale  maps  overlaid  with  operational  graphics. 
(NTC,  1994) 
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In  actuality,  recent  experience  at  the  NTC  shows  that  photographs  are  unavailable 
at  the  small  unit  level.  Platoon  AARs  are  held  in  the  vicinity  of  the  battlefield  within  30 
minutes  of  the  exercise’s  end.  Company  AARs  are  also  held  in  the  field  two  hours  after 
the  exercise’s  end.  The  photographic  aids  are  unavailable  for  these  AARs  because 
photographs  of  the  action  cannot  reasonably  be  taken,  developed,  and  delivered  in  time. 
The  terrain  model,  preprinted  text  slides,  dry-erase  boards,  and  sketch  pads  are  the 
primary  AAR  aids  used.  (Lartigue,  L.,  personal  communication,  2  November  1996,  and 
Lipinski,  M.,  personal  communication,  2  November  1996) 

CTC  AAR  System 

A  couple  of  observations  can  be  made  from  the  JRTC  and  NTC  AAR  guidance 
tapes.  First  and  foremost,  the  style  and  effectiveness  of  the  AAR  is  a  fimction  of  the 
AAR  leader.  The  AAR  leader’s  skill  is  critical  in  identifying  the  potential  discussion 
issues  and  then  deciding  what  format  is  best  for  drawing  out  those  issues  from  a 
particular  group  of  trainees.  The  AAR  leader  must  tailor  the  AAR  to  the  needs  of  the 
discussion  group.  As  pointed  out  at  the  end  of  Chapter  II,  this  is  dependent  upon  the 
collective  skill  level  of  the  unit. 

The  second  observation  is  that  three  of  the  determinants  of  an  effective  AAR  are 
verified  by  the  guidance.  Participation,  discussion  focus,  time  limits,  and  problems  and 
solutions  are  all  specifically  identified  as  important.  Learning  reinforcement  is  not 
addressed. 
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Given  the  AAR  guidance  of  the  CTCs  and  the  doctrinal  references,  the  conduct  of 
the  CTC  AAR  can  be  represented  as  a  system.  (Figure  3.1,  Conceptual  CTC  AAR 
System,  and  Figure  3.2,  CTC  AAR  Components)  This  is  only  a  part  of  the  AAR  process. 
The  entire  AAR  process  parallels  the  training  management  cycle  beginning  with  defining 
the  data  collection  and  feedback  requirements  from  the  training  objectives  and  purpose. 


OC 


Trainee 

Experience 


OPFOR 

Performance 

Observations 


*  Summary  of  AAR 
learning  points. 


Recorded 


Training 
Plan 


Take  Home  Package: 

*  Video  &  audio  record 
of  AAR 


Figure  3.1,  Conceptual  CTC  AAR  System 


The  inputs  into  the  AAR  are  collected  or  developed  during  training  exercise 
execution  and  AAR  preparation.  At  the  CTCs,  performance  data  is  primarily  collected 
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from  the  OC  team’s  and  the  OPFOR’s  perfonnance  observations.  These  observations  are 
hand  reeorded  by  the  OC  or  an  assistant. 

At  the  CTCs,  the  trainer  is  the  Division  Commander  of  the  brigade  participating 
in  the  training.  For  the  exercise,  the  Division  Commander  determines  the  training 
objectives,  missions  to  be  executed,  and  conditions  of  the  battlefield.  These  are  the  same 
conditions  that  are  reviewed  in  the  AAR  to  help  determine  what  happened,  why  it 
happened,  and  how  to  do  it  better.  The  Operations  Group  of  the  CTC  is  responsible  for 
implementing  the  training  plan,  controlling  the  exercise,  and  providing  feedback  at  each 
unit  level. 

The  fourth  input  to  the  system  is  recorded  performance  data.  Vehicles  and  key 
personnel  are  instrumented  so  that  position  and  movement  can  be  tracked  and  recorded 
throughout  the  exercise.  Additionally,  the  radio  transmissions  of  key  leaders  are  recorded 
and  time  stamped  to  aid  in  determining  what  happened  during  the  exercise.  Although 
these  are  not  available  during  the  AAR,  what  was  recorded  can  be  relayed  to  the  OC  via 
radio  communications.  Additionally,  the  MILES  kill  codes  are  recorded  by  each  weapon 
system  and  are  available  to  the  OC  team. 

The  last  input  is  the  training  experience  of  the  unit.  The  unit  experience  is  what  a 
unit  performs  during  training.  This  experience  is  formed  with  incomplete  knowledge  of 
the  situation  and,  as  a  result,  misperceptions.  Notwithstanding,  this  is  what  the  trainees 
know  when  they  come  into  the  AAR.  This  information  is  verbally  presented  by  the 
trainees  to  achieve  the  AAR  objectives. 
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The  outputs  of  the  CTC  AAR  process  are  a  summation  of  the  learning  points  from 
the  AAR  and  a  videotaped  record  of  the  AAR  itself.  The  OC  that  leads  the  AAR  is 
responsible  for  summarizing  the  learning  points.  The  trainees  write  these  points  in  their 
notes  or  may  obtain  a  text  copy  from  the  OC.  The  AAR  is  recorded  on  videotape  for  the 
unit.  This,  along  with  a  written  summary  of  the  OC  team’s  observations,  makes  up  the 
unit’s  take  home  package.  The  OPFOR  do  not  input  observations  directly  into  the  take 
home  package.  However,  the  OC  team  may  include  those  observations  in  the  written 
product. 

The  physical  and  conceptual  components  that  interact  during  the  AAR  are  shown 
in  Figure  3.2,  CTC  AAR  Components.  The  OC  leading  the  AAR  also  controls  the  AAR 
components.  He  controls  what  information  is  presented  when  and  on  which  medium. 

CTC  mediums  or  training  aids  include  terrain  models,  video  and  audio  display  systems, 
overhead  projectors,  chalkboard  or  sketch  pad,  pre-made  charts  with  textual  or  graphical 
data,  and  computer  display  systems.  The  OC  selects  the  medium  that  can  best  support  the 
learning  point. 

The  trainees  receive  feedback  from  the  AAR  leader,  OPFOR,  and  the  presentation 
mediums.  The  communication  between  the  trainees  and  the  AAR  leader/OPFOR  is  the 
discussion  between  them.  This  discussion  takes  the  form  of  questions,  answers,  and 
comments.  What  is  not  shown  is  the  communication  that  occurs  amongst  the  trainees. 

The  CTC  AAR  guidance  does  not  address  how  to  prompt  this  discussion  and  seems  to 
suggest  that  it  occurs  spontaneously. 


82 


Figure  3.2,  CTC  AAR  Components 


The  OPFOR  are  depicted  interacting  with  the  trainees.  In  fact,  this  occurs  only 
for  a  small  portion  of  the  AAR.  Because  of  training  exercise  considerations,  the  OPFOR 
representative  returns  to  his  unit  immediately  after  he  presents  his  observations  and 
answers  any  questions  the  trainees  may  have.  Usually,  this  lasts  for  approximately  10 
minutes.  (AAR  tapes  and  personal  experience) 
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The  AAR  recorder  is  responsible  for  the  audio  and  visual  record  of  die  AAR. 
Normally,  one  cameraman  films  the  AAR  from  a  stationary  point.  He  is  specifically 
responsible  for  capturing  the  key  points  and  the  discussion  during  the  AAR. 

The  leaming/teaching  method  component  is  addressed  by  the  NTC  AAR  guidance 
and  accounts  for  how  the  AAR  is  conducted/run.  The  NTC  guidance  identified  the 
teaching/lecture  method  as  appropriate  for  some  situations.  This  is  certainly  one  end  of 
the  leaming/teaching  method  spectrum.  For  the  CTCs,  the  other  end  of  the  spectrum  is 
defined  by  a  discussion  in  which  the  AAR  leader  and  trainees  participate.  The  amount 
and  distribution  of  the  participation  serve  to  define  the  leaming/teaching  method  states 
between  the  lecture  and  a  fully  participatory  AAR. 


Small  Unit  Simulation  Training  Systems 

Currently,  the  Army’s  small  unit  simulation  training  systems  are  SIMNET  and 
CCTT.  Both  train  platoon  and  company  collective  tasks  in  a  virtual  environment. 
SIMNET  is  the  legacy  system  and  CCTT,  its  replacement,  is  the  first  of  a  number  of 
virtual  training  systems  that  will  form  the  Combined  Arms  Tactical  Trainer  (CATT).  In 
completion,  the  CATT  will  consist  of  the:  CCTT  for  mechanized  infantry,  armor,  and 
cavalry  units;  Aviation  Combined  Arms  Tactical  Trainer  (AVCATT)  for  Army  aviation 
units;  Fire  Support  Combined  Arms  Tactical  Trainer  (FSCATT)  for  field  artillery  and 
mortar  units;  Air  Defense  Combined  Arms  Tactical  Trainer  (ADCATT)  for  air  defense 
units;  and  Engineer  Combined  Arms  Tactical  Trainer  (ENCATT)  for  engineer  units. 
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(STRICOM,  27  Aug  1996a)  Each  training  system  will  be  Distributed  Interactive 
Simulation  (DIS)  compliant  and  consist  of  networked,  high  fidelity,  manned  simulators, 
Semi-Automated  Forces  (SAF),  support  workstations,  computer  networks  and  protocols, 
and  an  AAR  system.  (STRICOM,  27  Aug  1996b) 

AAR  Systems  in  Training  Simulations 

Larry  Meliza’s  research  report.  Standardizing  Army  After  Action  Review  Systems 
(1996a),  is  the  most  complete  and  thorough  source  reviewing  Army  AAR  systems  and 
efforts  for  small  unit  training  systems  to  date.  Meliza  conducted  the  research  to  support 
development  of  the  needs  and  capabilities  for  the  Standard  Army  After  Action  Review 
System  (STAARS).  To  complete  this  research,  he  reviewed  the  relevant  literature, 
published  and  unpublished  findings  of  three  Army-wide  AAR  conferences,  and  examined 
two  ongoing  AAR  system  development  efforts  in  relation  to  a  legacy  AAR  system.  The 
Army  Training  and  Analysis  Feedback  System  (ATAFS)  and  the  Simulation  Training 
Integrated  Performance  Evaluation  System  (STRIPES)  were  compared  to  the 
development  of  the  Unit  Performance  Assessment  System  (UPAS)  to  clarify  the  needed 
capabilities  of  STAARS.  Many  of  Meliza’s  findings  were  used  by  the  National 
Simulation  Center  as  capability  specifications  and  rationale  for  the  STAARS  Operational 
Requirements  Document  (ORD).  (Meliza,  1996a,  viii) 

The  UPAS,  ATAFS,  STRIPES,  the  CCTT  AAR  system,  and  STAARS  are 
recoimted  in  detail  in  Appendix  C,  Current  AAR  Systems  in  Training  Simulations.  The 
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review  of  Meliza’s  research  reveals  that  these  systems  are  primarily  concerned  with 
reducing  the  workload  associated  with  preparing  the  AAR  and  formatting  the  information 
products  that  result  from  the  preparation.  In  other  words,  these  systems  focus  on 
automating  data  collection,  compilation,  and  display  to  reduce  the  amount  of  time  it  takes 
a  trainer/AAR  leader  to  prepare  an  AAR  for  presentation.  In  this  sense,  these  AAR 
systems  are  not  specifically  designed  to  improve  the  AAR’s  effectiveness  as  much  as 
they  are  to  ensure  its  timeliness. 

This  is  not  to  say  that  displaying  information  effectively  is/was  not  a  concern 
during  AAR  system  development.  The  initial  UPAS  development  effort  focused  on 
providing  information  displays  that  made  it  easier  for  trainees  to  grasp  what  happened 
during  a  training  exercise.  (Meliza  et  al.,  1992b)  This  resulted  in  the  five  format  displays 
incorporated  into  ATAFS. 

The  need  for  preparation  efficiency  is  driven  by  two  reasons.  As  Meliza  points 
out,  lower  echelon  AARs  precede  higher  echelon  AARs  so  that  the  results  of  the  first  can 
be  used  as  input  for  the  second.  Thus,  to  start  higher  echelon  AARs  in  a  timely  manner, 
lower  level  AARs  must  be  completed  as  soon  as  possible.  The  Mounted  Warfare 
Training  Simulation  Center  at  FT  BCnox,  KY  tries  to  begin  platoon  AARs  10  minutes 
after  the  end  of  a  training  exercise.  (Meliza,  1996a,  27)  Another  reason  to  rush 
preparation  is  that  xmits  may  need  to  conduct  subsequent  exercises  on  the  trainer  and 
therefore  need  the  AAR  system  to  collect  data  for  fiieir  exercise.  Currently,  none  of  the 
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AAR  systems  have  the  requirement  nor  capability  to  display  a  presentation  for  a  previous 
exercise  while  collecting  data  from  a  separate  running  exercise. 

None  of  the  systems  seek  to  promote  or  improve  trainee  participation  beyond 
information  presentation.  However,  well  designed  data  displays  will  aid  units  in 
determining  what  is  critical  and  what  happened.  The  more  quickly  ground  truth  can  be 
represented,  the  faster  the  unit  can  focus  on  why  it  happened  and  how  to  do  it  better. 
Except  for  STAARS,  none  of  the  systems  propose  tools  to  aid  in  determining  how  to 
perform  a  task  better.  STAARS  requires  the  capability  to:  “translate  lessons  learned 
from  the  CALL  [Center  for  Army  Lessons  Learned],  Battle  Command  Battle  Lab 
experiments,  CTC  rotations,  and  other  sources  into  leader  development  and  collective 
training  concepts,  methods  and  strategies  and  revised  doctrine  and/or  tactics,  techniques, 
and  procedures.”  (BCTD,  1996a,  para  3.a.2.a.6)  Although  this  capability  is  not  identified 
for  use  during  the  AAR,  it  is  conceivable  that  if  information  is  collected,  it  could  be 
stored  and  updated  for  use  in  the  AAR  also. 

The  retraining  of  specific  tasks  is  also  not  considered  essential  to  an  AAR  system 
capability.  Currently,  none  of  the  systems  require  this.  This  is  understandable  given  that 
the  popular  interpretation  of  the  training  model  paradigm  concludes  the  cycle  with  the 
AAR.  As  stated  before,  this  must  change  in  order  to  reinforce  the  learning  that  occurs 
during  the  AAR. 

The  bottom  line  of  these  simulation  AAR  subsystems  is  that  they  support  the 
implementation  of  the  CTC  AAR  system  and  will  reinforce  whatever  strengths  and 
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weaknesses  that  approach  has.  These  subsystems  contribute  directly  to  the  AARs 
effectiveness  by  reducing  the  required  preparation  time  and  collecting  a  wide  variety  of 
objective  performance  data.  Thus,  there  are  two  apparent  results.  First,  the  delay 
between  the  end  of  the  training  exercise  and  the  start  of  the  AAR  is  minimised.  And 
second,  performance  (what  happened)  can  be  more  accurately  and  objectively  described. 


Survey 

Purpose 

Given  that  the  CTC  AARs  are  what  the  training  community  and  research  and 
development  communities  are  trying  to  model,  how  effective  are  they  with  respect  to 
discussion  participation  and  focus?  What  factors  influenced  the  measures  of 
effectiveness  most?  The  purpose  of  the  survey  of  17  platoon  and  company  AARs  from 
the  JRTC  and  the  NTC  is  to  provide  indications  of  the  answers  to  these  questions.  A 
result  of  this  research  is  the  development  of  definitions  and  procedures  that  can  be  used  to 
assess  the  effectiveness  of  an  AAR.  Only  platoon  and  company  level  AARs  were  used 
because  the  focus  of  this  thesis  is  at  the  small  unit,  below  battalion  level. 

Measuring  Effectiveness 

The  measures  of  effectiveness  used  are  those  that  were  argued  in  Chapter  II  and 
supported  by  CTC  guidance  -  trainee  discussion  participation,  discussion  focus  on 
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learning  points,  and  time  length  of  AAR.  Since  follow-on  training  is  not  recorded  and 
there  is  no  indication  of  when  the  training  exercise  ended,  the  retraining  and  timeliness 
measures  of  effectiveness  cannot  be  measured  by  the  AAR  tape  alone.  In  any  case, 
immediate  retraining  is  not  permitted  at  the  CTCs  because  of  overall  scenario  time 
constraints.  These  same  time  constraints  drive  the  scheduling  of  all  AARs  and  the  OCs 
strictly  adhere  to  this  schedule.’  Hence,  the  timeliness  of  the  AAR  is  reasonably  constant. 
The  dependent  variables  that  combine  to  describe  each  measure  of  effectiveness  are 
discussed  below. 

Discussion  Participation  Dependent  Variables 

Trainee,  OPFOR,  and  AAR  leader  participation  allow  multiple  viewpoints  to  be 
considered  while  solving  problems  as  well  as  the  clear  and  complete  articulation  of  what 
happened  and  why  during  the  exercise.  To  collect  data  on  participation,  the  following 
were  counted: 

•  The  number  of  questions  and  comments  (utterances)  per  individual  during  the 

AAR; 

•  The  number  of  trainees  present  at  the  AAR; 

•  The  number  of  issues  identified  by  the  trainees  vs.  the  AAR  leader; 

•  The  number  of  training  solutions  identified  by  the  trainees  vs.  the  AAR  leader; 


’  AAR  schedule:  JRTC  -  platoon  AARs  are  conducted  1  or  2  hours  after  the  end  of  the  exercise 
(ENDEX),  and  company  AARs  are  conducted  4  hours  after  ENDEX;  NTC  -  platoon  AARs  are  conducted 
45  minutes  after  the  end  of  the  exercise,  and  company  AARs  are  conducted  90  minutes  after  ENDEX. 
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•  Occurrence  of  discussion  amongst  the  unit  without  AAR  leader  intervention. 
The  number  of  utterances  per  individual  and  the  total  number  of  trainees  were  the  same 
data  collected  by  Downs  et  al.  (1987).  Displayed  as  ratios  of  unit  to  AAR  leader 
utterances,  these  will  indicate  a  relative  level  and  distribution  of  participation  amongst  the 
AARs  surveyed.  Tracking  the  number  of  questions  and  comments  will  give  some 
indication  of  active  participation  by  the  trainees  and  the  use  of  the  questioning  technique 
by  the  AAR  leaders.  A  ratio  of  unit  comments  to  AAR  leader  questions  vdll  indicate  how 
many  units  made  comments  beyond  the  question  responses  to  the  AAR  leader. 

For  this  research,  an  issue  is  defined  as  a  subject  or  point  for  discussion.  The 
issue  can  be  specifie  or  general  but  must  pertain  to  the  unit’s  performance.  Thus,  a 
specific  task  such  as  control  movement  of  a  platoon  or  a  general  subject  such  as  planning 
are  counted  as  issues.  The  term  “key  issues”  is  used  extensively  in  AARs  throughout  the 
Army  and  is  understood  to  be  synonymous  with  the  points  that  will  be  discussed.  Either 
the  AAR  leader,  unit  commander,  or  AAR  participants  can  decide  what  issues  are 
important  enough  to  spend  discussion  time  on. 

The  questions  counted  were  interrogative  statements  or  fi'agments  that  were 
intended  for  response.  Rhetorical  questions  were  coimted  as  comments.  The  comments 
counted  were  sentences,  phrases,  or  utterances  in  response  to  a  question  or  another 
comment.  Rephrased  questions  or  comments  were  counted  individually  so  that  a  posed 
question  and  the  re-worded  elaboration  to  explain  the  question  count  as  two  or  three 
utterances.  Repeated  questions  or  comments  were  not  counted  separately  unless  they 
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were  uttered  at  least  five  seconds  apart.  This  collection  constraint  was  implemented  in  an 
attempt  to  discoimt  a  high  count  of  utterances  from  an  individual  who  has  a  habit  of 
immediately,  and  unnecessarily,  repeating  himself 

Identifying  who  submits  problems  and  solutions  for  discussion  was  also  tracked. 
This  indicates  how  successful  the  AAR  leader  is  at  drawing  learning  points  (discussed 
below)  out  of  the  unit.  Basically,  this  is  a  part  of  guiding  the  discussion.  The  assumption 
is  that  units  will  discuss  the  problems  that  they  identify  more  readily  than  those  that  are 
dictated  to  them  by  the  AAR  leader. 

Discussion  Focus  Dependent  Variables 

As  outlined  in  Chapter  II,  the  complete  learning  point  (problem  space)  consists  of 
a  clearly  defined  issue  and  a  practical  solution.  A  clear  issue  in  Army  training  terms  is 
the  identification  of  the  doctrinal  training  task  that  was/was  not  performed  lAW  doctrinal 
performance  standards.  The  specific  performance  measures  that  define  the  desired 
outcome  must  be  identified.  Otherwise,  it  will  not  be  clear  what  portion  of  task 
performance  must  be  improved.  An  example  of  this  could  be  identifying  Perform 
Tactical  Road  March  (DA,  ARTEP  7-8-MTP,  1994,  5-82  to  5-84)  as  a  problem  issue. 
Without  further  explanation  of  the  five  task  standards  or  seven  associated  subtask 
standards,  there  is  no  way  to  tell  what  was  specifically  wrong  with  the  collective 
performance  of  this  task.  To  clarify  the  issue,  the  problem  must  be  articulated  in  the 
following  terms:  task  standard  #4  -  “the  main  body  was  not  surprised  by  the  enemy”  was 
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failed  because  subtask  #4  -  “the  platoon  maintains  local  security  throughout  the 
movement”  was  not  performed  correctly.  A  problem  statement  is  clear  only  if  it  is  stated 
in  terms  of  a  doctrinal,  collective/individual  training  task/drill.  This  is  a  correct  problem 
statement  if  it  results  in  the  correction  of  the  original  outcome.  Thus,  if  the  platoon 
corrects  its  actions  concerning  security  during  movement,  then  the  enemy  will  not 
surprise  the  unit’s  main  body.  If  this  does  not  correct  the  outcome  of  the  unit  being 
surprised,  then  a  different  problem  task  must  be  found. 

Problems  that  were  not  specified  in  the  doctrinal  structure  outlined  above  were  not 
coimted.  The  reason  for  this  is  that  the  Army  training  system  (Chapter  I)  is  designed  to 
correct  problems  that  conform  to  this  structure.  This  is  a  basic  premise  of  the 
standardization  of  battle  focused  training  across  the  Army.  (DA,  FM  25-101, 1990) 
Without  a  standardized  training  process  the  Army  does  not  have  a  common  language  to 
communicate  and  correct  performance  problems.  Thus,  to  avoid  subjective  interpretation 
of  message  content,  issue  and  solution  tasks  and  task  standards/performance  measures 
were  only  counted  if  expressed  in  terms  of  doctrinal  training  terms.  Likewise,  conditions 
for  the  future  training  plan  had  to  be  articulated  with  respect  to  METT-T  or  other 
doctrinal  terms. 

The  BDA  from  an  engagement  is  used  to  determine  the  correct  root  problem  and 
“why”  it  was  a  problem.  In  the  above  case,  the  BDA  would  need  to  be  examined  to 
determine  if perform  local  security  was  a  problem  for  the  unit.  The  following  BDA 
would  be  relevant  to  the  articulation  of  the  problem: 
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•  Who  saw  the  other  first; 


•  Who  shot  first; 

•  Length  of  time  required  to  place  controlled  fire  on  the  enemy;  and 

•  The  nmnber  of  casualties  and  equipment  losses  that  occurred  before  and  during  the 

initial  engagement. 

The  BDA  will  reveal  the  root  training  problem  via  cause-effect  analysis.  The  number  of 
casualties  and  equipment  losses  are  the  consequence  of  the  unit’s  actions  or  inaction. 

This  information  signifies  the  elements  of  performance  that  contribute  most  directly  to 
the  outcome. 

The  solution  should  then  describe  how  task  performance  is  to  be  corrected  or 
improved  in  terms  of  the  training  conditions  and  any  other  related  tasks  and  subtasks  that 
affect  the  stated  performance  measures.  Using  the  example  above,  the  solution  must 
identify  the  actions  (supporting  subtasks)  required  to  maintain  security  during  movement 
under  the  specified  training  conditions.  The  training  conditions  are  identified  in  terms  of 
the  unit’s  and  enemy’s  mission,  intent,  and  organization;  the  terrain  considerations;  and 
die  time  available  for  planning,  preparation,  and  execution.  Given  the  training 
conditions,  the  supporting  subtasks  and  associated  performance  measures  to  achieve  local 
security  make  up  a  clear  solution.  This  data  variable  is  labeled  solutions  developed. 

The  next  step  is  to  articulate  how  to  train  the  root  problem  tasks  in  the  future. 
This  includes  the  identification  of  the  necessary  training  conditions  and  performance 
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measures  that  must  be  met  in  retraining.  The  result  is  a  training  plan  —  the  task, 
conditions,  and  standards  that  must  be  performed  and  met  to  correct  the  weakness.  These 
are  labeled  solutions  planned.  The  difference  between  a  solution  developed  and  a 
solution  planned  is  the  context  to  which  each  applies.  The  solution  developed  concerns 
corrective  actions  that  are  required  to  affect  performance  improvement  in  the  training 
scenario  just  conducted.  A  solution  planned  pertains  to  the  corrective  actions  required  to 
ensure  the  weakness  is  not  repeated  (during  task  execution)  given  any  set  of  battlefield 
conditions. 

For  a  solution  planned  Xo  be  coimted,  it  must  meet  a  feasibility  criterion.  This 
criterion  is  that  the  prescribed  retraining  task,  conditions,  and  standards  must  be  feasible 
enough  for  unit  leaders  to  implement  with  the  resources  available  to  them.  This  translates 
to  the  appropriate  amount  of  time,  logistical  support,  training  system,  and  land  being 
available  at  the  unit’s  home-station.  Subject  matter  experience  indicates  that  most  unit 
members  automatically  limit  themselves  to  the  resources  immediately  available  when 
planning  training;  exceeding  those  resources  is  often  unimaginable  for  them. 

An  example  of  a  solution  planned  to  correct  the  problem  of  perform  local  security 
for  the  collective  task  Perform  Tactical  Roadmarch  is  as  follows.  Assume  ftiat  a  platoon 
was  moving  in  the  least  secure  vehicular  movement  formation  at  the  time  of  the  surprise 
enemy  attack.  A  viable  training  solution  may  be  to  train  leaders  on  a  terrain  model  to 
analyze  the  enemy  situation  with  respect  to  the  terrain  in  order  to  predict  likely  enemy 
ambush  positions.  This  is  reasonable  because  combat  units  have  access  to  a  terrain  model 
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most  of  the  time  and  it  takes  few  resources  to  coordinate  and  execute  the  training.  An 
unreasonable  training  solution  would  require  resources  that  the  platoon  leadership  does 
not  control. 

To  measure  the  quality  of  the  learning  points,  the  following  were  counted: 

•  The  number  of  solutions  planned  —  problem  tasks  described,  solved,  and 

planned  for  future  training  with  feasible  task,  conditions,  and  standards; 

•  The  number  of  solutions  planned  with  task,  conditions,  and  standards  that  are 

not  feasible  with  respect  to  training; 

•  The  number  of  solutions  developed — problem  tasks  described  and  solved 

without  corrective  training  actions  articulated; 

•  The  number  of  problem  tasks  identified  without  adequate  description  of 

training  conditions  and  performance  standards  or  any  solution  at  all; 

•  The  number  of  ill-defined  issues  (abstract  problems  or  solutions  not  associated 

with  a  doctrinal  task  or  performance  standard). 

Of  course,  the  number  of  issue  tasks  identified  with  a  feasible  retraining  plan  should  not 
be  a  large  number.  Doctrinal,  JRTC,  and  NTC  guidance  recommend  that  two  to  four 
learning  points  be  fully  developed  during  the  AAR.  (DA,  TC  25-20, 1993,  JRTC/BDM, 
1993,  and  NTC/BDM,  1992)  Therefore,  the  number  of  solutions  planned  and/or 
developed  is  not  as  important  as  the  fact  that  there  were  some  solutions  planned  and/or 
developed  rather  than  none. 
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Learning  occurs  in  the  participant’s  articulation  and  elaboration  during  the 
discussion.  The  participant,  rather  than  the  AAR  leader,  must  articulate  and  elaborate  on 
the  task  specific  strengths,  weaknesses,  and  factors  that  influenced  performance  and  then 
develop  a  retraining  plan  to  correct  or  reinforce  behavior.  For  this  reason,  the  tracking  of 
fully  developed  issues  and  solutions  is  an  objective,  albeit  relative,  measure  of  unit 
learning  that  occurred  during  the  AAR.  But  if  the  participant  can  verbally  articulate  the 
subject,  then  there  is  a  strong  probability  that  he  knows  the  subject.  Hence,  a  unit  that 
articulates  a  number  of  planned  and  developed  solutions  has  a  better  chance  to  improve 
performance  than  a  unit  that  does  not.  Furthermore,  the  unit  will  collectively  accept  the 
solution  and  be  more  likely  to  implement  it  if  it  was  developed  by  a  unit  member  rather 
than  an  outsider.  Unit  members  are  more  credible  to  each  other  because  they  are  familiar 
with  the  organizational  climate,  resources,  and  problems,  and  have  to  live  with  or  share 
the  solution’s  consequences  with  the  others. 

Time  Dependent  Variables 

Time  is  the  final  measure  of  effectiveness  considered  in  this  survey.  Each  CTC 
sets  AAR  time  limits  that  are  given  to  the  unit  prior  to  the  AAR.  Unless  the  AAR  leader 
states  otherwise,  the  trainees  expect  the  AAR  to  last  no  longer  than  the  allotted  time. 
Hence,  they  vuider stand  that  they  must  remain  attentive  throughout  the  AAR.  For  the 
JRTC,  platoon  and  company  AARs  are  allotted  60  and  120  minutes  respectively.  At  the 
NTC,  platoon  and  company  AARs  are  allotted  45  and  90  minutes  respectively.  For  this 
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survey,  it  will  be  noted  whether  or  not  the  AAR  exceeds  the  designated  time  limit.  The 
number  of  trainees  sleeping  during  the  AAR  can  also  indicate  that  the  AAR  is  too  lengthy 
or  that  the  AAR  leader  is  unskilled  at  soliciting  participation. 

The  length  of  time  spent  on  each  stage  of  the  AAR  was  also  recorded.  The  AAR 
stages  correspond  to  the  AAR  agenda  followed  by  each  AAR  leader.  Although  specific 
agenda  items  vary  between  AARs,  they  generally  follow  the  doctrinal  guidance  outlined 
in  TC  25-20.  (Table  3.3,  Doctrinal  AAR  Format)  The  allocation  of  time  in  accordance 
with  the  agenda  shows  how  much  of  the  AAR  time  (in  minutes)  was  used  for 
administrative  topics,  introduction  topics,  discussion,  and  conclusion  topics.  Many  of  the 
introduction  and  conclusion  topics  are  required  by  doctrine.  The  number  of  questions  and 
comments  by  each  participant  was  recorded  from  items  4, 5,  and  6  of  die  doctrinal  AAR 
format.  These  items  are  the  discussion  of  key  issues,  discussion  of  optional  issues,  and 
discussion  of  force  protection  issues.  In  the  other  sections  of  the  AAR,  doctrinal 
guidance  does  not  require  participatory  discussion.  (DA,  TC  25-20,  1993)  The  limits  of 
the  discussion  period  were  defined  by  the  AAR  leader  and  the  agenda.  Thus,  when  the 
AAR  leader  moved  from  the  summary  of  recent  events  to  the  discussion  of  key  issues,  the 
discussion  period  began.  The  discussion  period  ended  when  the  AAR  leader  began  the 
closing  comments  and  summary  portion.  It  was  also  qualitatively  noted  if  the  AAR 
leader  followed  the  doctrinal  AAR  format  or  created  his  own. 
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Table  3.3,  Doctrinal  AAR  Format 


1 .  Introduction  and  rules.  The  AAR  leader  describes  the  purpose  and  sequence  of  the 
AAR  emphasizing  that  “the  AAR  is  not  a  critique”  and  all  should  participate  in  the 
discussion  to  improve  performance. 

2.  Review  training  objectives  and  intent.  The  training  objectives,  friendly  and  OPFOR 
commander’s  mission  and  intent,  and  relevant  doctrine,  tactics,  techniques,  and 
procedures  are  reviewed. 

3.  Summary  of  recent  events.  Using  leading  and  open-ended  questions,  the  AAR 
leader  guides  a  discussion  of  the  logical  sequence  of  events. 

4.  Discussion  of  key  issues.  The  AAR  leader  uses  a  discussion  technique  to  help 
participants  discover  strengths/weaknesses,  develop  solutions,  and  designate 
corrective  action.  The  three  discussion  technique  structures  available  are:  a 
chronological  order  of  events,  the  seven  Battlefield  Operating  Systems  (BOS),  and 
key  events/themes/issues. 

5.  Discussion  of  optional  issues.  The  AAR  leader  has  the  option  to  lead  the  discussion 
to  the  topics  of  soldier/leader  skills,  tasks  to  sustain/improve,  statistics,  or  other 
topics. 

6.  Discussion  of  force  protection  issues.  This  is  a  mandatory  topic.  It  covers  all 
aspects  of  soldier  safety  in  the  field  and  in  garrison  and  is  emphasized  throughout 
the  AAR. 

7.  Closing  comments  and  summary.  The  AAR  leader  reviews  and  summarizes 
learning  points  from  the  discussion  and  future  training  tasks  and  conditions. 

(DA,  TC  25-20,  1993, 1-3  and  Chapter  4) 


AAR  Leader  and  Unit  Actions  -  Independent  Variables 

Nine  other  qualitative  observations  made  during  the  survey  are  effectiveness 
indicators  identified  in  the  literature  review.  These  observations  are  listed  below  and  then 
discussed. 
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•  Introduction  establishing  a  non-threatening,  participatory  climate  (Downs  et  al., 

1987); 

•  Clear  review  of  training  objectives  and  executed  plan  (SHERIKON,  1996); 

•  Use  of  specific,  non-qualitative  performance  feedback  (SHERIKON,  1996); 

•  Use  of  a  3-D  terrain  model/representation  (Word,  1987); 

•  Depict  performance  results  with  pictures,  sketches,  and  graphs  (SHERIKON, 

1996); 

•  OPFOR  or  OPFOR  representative(s)  participating  in  the  discussion  (Word, 

1987). 

As  pointed  out  in  Chapter  II,  each  of  these  contribute  to  participation,  discussion  focus,  or 
both.  The  last  three  observations  are  also  factors  that  could  possibly  affect  die  unit’s 
participation  in  the  discussion. 

•  Unit  attitude; 

•  Unit  leader’s  actions  to  support  a  non-threatening,  participatory  climate. 

•  Participation  feedback. 

Downs  et  al.  found  that  the  AAR  leader’s  emphasis  of  cooperation,  participation, 
and  open  discussion  in  the  introductory  remarks  had  a  strong  effect  on  participation. 
(Chapter  II,  30)  In  this  survey,  it  was  noted  whether  or  not  the  AAR  leader  addressed  this 
issue  in  the  introduction  or  at  any  other  time  during  the  AAR.  Examples  of  these 
statements  are: 

“The  focus  is  on  how  to  improve,  not  who  to  blame,  be  objective;” 
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“It  is  okay  to  disagree  with  a  point  being  made;” 

“This  is  your  AAR,  your  discussion  is  important  to  determining  what  was  good 
or  bad  about  your  performance,  you  decide;” 

“The  discussion  is  how  we  learn  from  what  happened  and  I  (the  AAR  leader)  am 
here  to  learn  also.  I  leam  something  new  in  each  AAR  I  conduct.” 

This  is  not  a  complete  list,  but  these  statements  are  examples  of  what  the  AAR  leader 
should  be  trying  to  commimicate  to  meet  the  requirements  of  Downs’  et  al.  (1987) 
conclusion.  The  AAR  leader’s  statements  were  recorded  for  the  survey  and  then 
transposed  as  qualitative  data.  If  the  AAR  leader  made  similar  statements,  the  data  point 
was  marked  as  a  positive  one  (+1).  If  he  did  not  make  any  statements  of  this  type,  the 
data  point  was  marked  as  a  negative  one  (-1). 

SHERIKON  stated  that  a  clear  review  of  the  training  objectives  and  executed  plan 
was  important  to  AAR  effectiveness.  (Chapter  II,  42)  This  is  important  from  the 
standpoint  that  it  allows  trainees  to  compare  what  should  have  happened  according  to  the 
plan  (training  objectives)  with  what  actually  happened.  For  data  collection,  it  was  noted 
if  the  unit’s  plan  was  reviewed  and  if  the  comparison  was  made. 

SHERIKON,  Word,  and  Downs  et  al.  all  identified  the  need  for  specific,  non- 
qualitative  performance  feedback  or  BDA.  (Chapter  II,  39)  The  most  popular  data  used 
for  this  purpose  is  the  number  of  casualties  and  equipment  losses  inflicted  and  received. 
For  each  AAR,  it  was  noted  whether  or  not  the  BDA  was  consolidated  and  presented 
during  the  AAR. 
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Word  (1987)  was  the  strongest  advocate  of  the  next  observation  -  whether  or  not 
the  AAR  leader  uses  a  3-D  representation  of  the  terrain  during  the  AAR.  (Chapter  II,  29) 
The  terrain  model  helps  visualization  of:  what  happened  and  why,  terrain  considerations, 
and  xmderstanding  singular  actions  in  terms  of  the  larger  picture  of  events.  Consequently, 
it  aids  participants  in  communicating  their  discussion  points.  It  was  noted  if  a  terrain 
model  was  present  and  if  it  was  used  in  each  AAR. 

SHERIKON  and  Word  advocate  the  use  of  pictures,  sketches,  and  graphs  to 
depict  performance  results  for  the  same  reasons  as  stated  above.  Additionally,  a  number 
of  studies  show  that  humans  can  comprehend  pictures,  graphs,  and  tables  much  quicker 
than  they  do  written  or  spoken  words.  (Chapter  II,  42)  For  each  AAR,  it  was  noted 
whether  or  not  any  of  these  presentation  formats  or  aids  were  used. 

The  next  observation  made  in  the  survey  was  whether  or  not  the  OPFOR  or 
member  of  the  OPFOR  participated  in  the  AAR  and/or  discussion.  Word  and  Bosley  et 
al.  pointed  out  the  importance  of  OPFOR  participation  in  the  AAR.  (Chapter  II,  28  and 
29)  Their  actions  largely  determine  what  the  friendly  unit  saw  and  acted  upon  during  the 
training.  Hence,  they  are  important  for  identifying  and  explaining  the  factors  that  caused 
the  performance  outcome.  They  also  lend  a  new  set  of  viewpoints  and  experience  to  the 
discussion  of  how  to  do  things  better. 

Another  item  noted  during  the  survey  was  whether  the  unit  was  hostile, 
argumentative,  or  defensive  in  attitude  toward  the  AAR  leader  or  each  other.  Examples 
of  these  attitudes  are  arguing  and/or  extensive  complaining  about:  exercise  rules  of 
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engagement,  OC  actions,  OPFOR  actions  considered  unfair,  and/or  mission  constraints 
imposed  by  the  higher  level  of  command.  Although  it  is  impossible  to  determine  why  the 
trainees  may  have  such  an  attitude,  it  does  detract  from  discussion  participation  and 
focus.  Not  much  learning  occurs  when  participants  are  personally  attacking  the  AAR 
leader  or  are  closed-minded  to  differing  opinions.  These  comments  were  noted  as 
administrative  utterances  and  combined  to  indicate  a  defensive/argumentative,  neutral,  or 
positive/proactive  learning  attitudes. 

The  next  research  observation  made  during  the  AAR  discussions  was  of  the  unit 
leader’s  actions.  Specifically,  whether  or  not  the  unit  leader  helped  to  prompt 
participation  amongst  the  unit  and  focus  the  discussion  on  the  problem  issues.  The  unit 
leader  can  play  a  large  part  in  curbing  a  hostile  and/or  defensive  attitude(s).  Essentially, 
he  has  the  choice  of  actively  facilitating  the  AAR,  hindering  the  AAR,  or  ignoring  the 
opportunity  to  influence  the  AAR  at  all.  The  unit  leader’s  explicit  comments  related  to 
this  choice  were  noted  as  administrative  utterances.  Comments  or  open-ended  questions 
that  were  related  to  the  discussion  are  noted  as  discussion  utterances. 

The  last  discussion  observation  is  the  AAR  leader’s  provision  of  participation 
feedback  to  the  trainees.  From  subject  matter  experience  it  seems  that  declarative 
remarks  that  laud  participation  and/or  highlight  non-participation  prompt  the  trainees  to 
participate  in  the  discussion.  Examples  of  theses  remarks  are: 

“You  guys  need  to  talk  this  out,  what  do  you  think  about  this  (problem)?” 

“Good  point,  what  do  the  rest  of  you  say?  Do  you  agree?” 
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“You’re  not  participating  in  the  discussion,  do  you  think  what  he  said  is  true? 
Why  (or  why  not)?” 

“That  is  a  valid  point,  good  job  on  pointing  it  out!” 

“You  guys  are  not  contributing;  they  are  talking  about  a  problem  that  affects  you. 
Say  whatyoM  think.” 

“Good  job  (everyone)  on  discussion;  keep  it  up.” 

These  remarks  serve  to  remind  the  trainees  that  discussion  participation  is  an  objective  of 
the  AAR.  They  are  also  classic  examples  of  discussion  facilitation  -  one  of  the  AAR 
leader’s  primary  responsibilities.  (DA,  FM  25-101,  1990,  and  TC  25-20,  1993)  Oddly 
enough,  providing  participatory  feedback  is  only  implied  in  the  doctrinal  references. 
Open-ended  and  leading  questions  are  specifically  designated  as  facilitation  techniques. 


Methodology 

The  JRTC  and  NTC  were  solicited  for  platoon  and  company  level  tapes  of  actual 
AARs.  The  tape  selection  criteria  for  each  training  center  was  as  follows:  select  five 
each  platoon  and  company  AAR  tapes,  select  from  rotations  that  occurred  in  the  past  few 
years,  select  AARs  led  by  different  AAR  leaders,  select  AARs  from  infantry  or  armor 
units,  and  select  AARs  from  different  imits.  The  result  of  this  request  was  the  receipt  of  9 
platoon  and  7  company  level  AAR  tapes  from  the  CTCs.  An  additional  company  level 
JRTC  AAR  from  my  personal  library  was  also  included  for  a  total  of  17  AARs.  The 
AARs  were  all  conducted  within  a  recent  15  month  span.  None  of  the  AARs  were 
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conducted  by  the  same  AAR  leader  and  all  were  of  different  platoon  and  company  units. 
The  type  of  each  unit  is  listed  in  Table  3.4,  AAR  Survey  -  Unit  Types.  For  the  platoon 
AARs,  attendance  ranged  from  16  to  45.  Company  level  AARs  involved  only  key 
leaders  and  the  company  headquarters  section  so  attendance  ranged  from  7  to  16 
depending  upon  the  number  of  attached  units. 


Table  3.4,  AAR  Survey  -  Unit  Types 


Echelon 

Type 

Number 

Platoon 

Infantry,  Light 

6 

Platoon 

Infantry, 

Mechanized 

2 

Platoon 

Armor 

2 

Company 

Infantry 

4 

Company 

Armor 

1 

Company 

Infantry-Armor 

Team 

I 

Company 

Armor  Heavy  Team 

I 

The  survey  of  17  AAR  tapes  was  conducted  over  a  14  day  period.  Except  for  4 
days,  1  tape  was  surveyed  per  day.  The  data  collection  was  conducted  by  two  observers, 
one  watched  and  listened  to  the  AAR  leader  while  the  second  concentrated  on  the  other 
participants.  Two  JRTC  platoon  AARs  that  were  not  included  in  the  survey  were  used 
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for  data  collection  practice.  This  allowed  the  data  collectors  to  proceduralize  the 
collection  process  before  beginning  the  survey. 

Since  the  AAR  leaders  dominated  the  discussions  in  the  majority  of  AARs,  the 
observer  attending  (listening  and  watching)  to  the  trainees  made  notes  of  discussion  focus 
and  other  observations  (agenda,  time  records,  effectiveness  indicators,  and  umt  attitude). 
This  person  is  an  active  duty.  Infantry  Major  with  over  seven  years  troop  experience  and 
two  years  CTC  experience  at  the  JRTC.  The  priority  of  effort  for  data  collection  was  to 
participatory  comments  first,  then  to  discussion  content,  and  lastly  to  the  other 
observations.  When  the  pace  of  recording  both  participatory  and  discussion  data 
prevented  collection  of  the  tertiary  priority  data,  the  AAR  was  viewed  a  second  time  in 
order  to  collect  the  data  on  the  other  observations. 

As  stated  before,  the  second  data  collector  counted  the  number  and  type  of 
utterances  the  AAR  leader  made.  This  observer  has  a  Bachelor  of  Science  in  Sports 
Journalism  with  ten  years  writing  and  reporting  experience  in  the  fields  of  sports, 
education,  and  marketing.  Additionally,  she  is  familiar  with  the  military  vernacular,  as 
well  as  military  terms  and  symbols. 

Analysis 

The  data  from  each  AAR  was  recorded  on  preprinted  forms  and  then  transcribed 
to  a  spreadsheet  for  analysis.  The  small  number  of  AARs  allowed  the  raw  data  to  be 
consolidated  into  two  tables.  The  aim  of  this  analysis  was  to  determine  what  factors  and 
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variables  seem  to  affect  participation  and  discussion  focus.  First,  a  trend  analysis  of  each 
factor  was  conducted.  This  followed  the  same  lines  of  inquiry  that  were  followed  in  the 
analysis  of  the  Downs  et  al.  (1987)  data.  (Chapter  II) 

The  second  part  of  the  analysis  was  to  use  univariate  multiple  regression  to 
confirm  any  trends  found.  The  regression  analysis  was  conducted  with  MINITAB  for 
Windows  software.  (Appendix  B,  Regression  Analysis)  All  regression  models  were 
tested  to  a  statistical  significance  level  (a)  of  0.05.  Multiple  regression  analysis  was 
conducted  for  each  of  the  dependent  variables  using  the  least  squares  method.  (Scheaffer 
&  McClave,  1995, 541  through  590)  This  analysis  indicated  which  factors  (AAR  leader 
and  unit  actions)  were  statistically  significant  and  seemed  to  have  a  large  or  small  effect 
on  a  selected  dependent  variable.  First,  a  dependent  variable  is  modeled  with  all  of  the 
independent  variables.  Then,  insignificant  independent  variables  are  individually 
removed  from  the  full  model.  The  insignificance  of  a  variable  is  determined  by  a  F-test 
achieved  (at  a  =  0.05).  The  objective  in  this  process  is  to  minimize  the  sixmmed  squares 
of  error  and  maximize  the  model’s  coefficient  of  determination  (R^,  a  measurement  of 
model  adequacy).  The  removal  of  variables  continues  until  a  reduced  model  of  least 
squares  is.  The  multicollinearity  amongst  the  regressor  variables  was  tested  using  eigen 
analysis.  (Myers  &  Montgomery,  1995, 656  through  662)  For  each  model,  the  variance 
inflation  factor  (VIF)  for  the  smallest  eigenvalues  was  computed.  A  VIF  >100  was 
considered  large  and  an  indication  of  multicollinearity.  The  regressors  that  did  not  pass 
the  test  were  reanalyzed  and  either  combined  or  removed  from  the  model  altogether. 
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Data 


The  quantitative  dependent  variables  selected  to  describe  the  factor  of  discussion 
participation  are  listed  below.  A  consolidated  list  of  variables  and  their  definitions  is  in 
Appendix  B,  Regression  Analysis,  Table  B-1,  Factor  Definitions. 

•  Unit: AAR  leader  utterance  ratio  -  total  unit  utterances  to  total  AAR  leader 
utterances. 

•  Percent  (%)  unit  participation  -  number  of  unit  discussion  participants  over 
total 

number  of  unit  attendees. 

•  Unit  comment:AAR  leader  question  ratio  -  total  number  unit  comments  to 
total  AAR 

leader  questions. 

•  Number  of  questions  asked  by  the  unit  during  the  discussion. 

•  Niunber  issues  discussed  amongst  unit  members  without  the  AAR  leader’s 
prompting 

or  intervention. 

The  observations  for  each  AAR  are  listed  in  Table  3.5,  Discussion  Participation  Data. 

The  dependent  variables  that  describe  discussion  focus  were  solutions  planned 
and  solutions  developed.  These  two  measures  were  further  subdivided  between  Unit 
specified  solutions  -  the  number  planned  and  developed  by  the  unit,  and  Total  specified 
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solutions  —  the  number  planned  and  developed  by  both  the  unit  and  the  AAR  leader 
together.  This  data  is  listed  in  Table  3.6,  Discussion  Focus  Data. 

The  AAR  leader  actions  that  are  hypothesized  to  affect  the  effectiveness  of  the 
AAR  are  shown  in  Table  3.7,  AAR  Leader  Actions  Data,  and  Table  3.8,  Unit  and  Other 
Actions  Data.  These  actions  are  qualitatively  coded  -  either  they  were  or  were  not 
performed.  The  code  for  an  action  performed  is  +1  (positive  one)  and  die  code  for  not 
performed  is  -1  (negative  one);  a  0  (zero)  code  represented  a  level  between  +1  and  -1.* 
The  introduction  establishing  a  participatory  climate,  unit  attitude,  and  unit  leader 
facilitating  discussion  were  all  actions  that  used  three  qualitative  levels.  For  example,  the 
introductory  comments  that  warrant  a  “0”  code  are  ones  that  are  stated  quickly,  from  rote 
memory,  or  read  from  a  preprinted  card.  Only  comments  that  added  emphasis  to  a  point 
were  given  a  code  of +1.  Likewise,  unit  attitude  and  unit  leader  facilitation  were  coded 
with  a  0  if  they  were  neither  positive  nor  negative  in  content. 


*  For  the  regression  analysis,  all  variables  were  represented  as  two  coded  levels,  “1”  and  “0.”  When  a 
third  condition  had  to  be  represented,  an  additional  two  level  variable  was  added. 
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Table  3.5,  Discussion  Participation  Data 
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Table  3.6,  Discussion  Focus  Data 


no 


Table  3.7,  AAR  Leader  Action  Data 


Table  3.8,  Unit  and  Other  Actions  Data 


AAR# 

Unit 

Attitude 

Unit 

Leader 

Aided 

discussion 

OPFOR 
Participation 
in  discussion 

1 

0 

0 

-1 

2 

-1 

-1 

-1 

3 

0 

0 

-1 

4 

-1 

0 

-1 

5 

0 

0 

-1 

6 

0 

0 

-1 

7 

-1 

0 

-1 

8 

0 

-1 

-1 

9 

0 

0 

-1 

10 

-1 

0 

0 

11 

0 

0 

-1 

12 

1 

1 

0 

13 

0 

0 

-1 

14 

1 

1 

-1 

15 

-1 

-1 

-1 

16 

0 

-1 

-1 

17 

-1 

-1 

-1 

Results 

Findings 

The  findings  of  this  survey  are  summarized  below.  Of  the  17  AARs,  4  were 
found  to  have  a  high  degree  of  trainee  participation  in  the  discussion  and  were  discussion 
focused.  In  the  summary  below,  the  “majority  of  AAR  leaders”  or  “AARs”  refers  to  the 
13  that  did  not  achieve  high  participation  and  specify  problem  solutions.  The  reciprocal 
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of  this,  the  “minority”  of  AARs  or  AAR  leaders,  refers  to  the  4  participatory  and 
discussion  focused  AARs  that  did  specify  performance  problem  solutions. 

•  All  17  AAR  leaders  followed  the  doctrinal  AAR  format  outlined  in  TC  25-20 
(1993).  (Table  3.3,  Doctrinal  AAR  Format) 

•  In  the  majority  of  AARs,  unit  member  participation  in  the  discussion  is  low 
relative  to  the  AAR  leader. 

•  The  majority  of  AAR  leaders  are  either  unskilled  at  using  open-ended  and 
leading  questions  or  do  not  use  them  at  all. 

•  The  majority  of  AAR  leaders  do  not  ensure  that  all  of  the  unit  members  are 
involved  in  the  discussion. 

•  The  majority  of  AAR  leaders  do  not  provide  participatory  feedback  during  the 
discussion. 

•  The  majority  of  AAR  leaders  do  not  use  doctrinal  performance  standards  nor 
specific  performance  feedback  to  focus  discussion. 

•  OPFOR  representatives  and  3-D  terrain  models  are  not  used  to  prompt,  focus, 
nor  facilitate  AAR  discussion. 

•  The  majority  of  AAR  leaders  and  units  do  not  summarize  nor  express 
strengths/weaknesses  (lessons  learned,  or  issues  discussed)  in  terms  of  a  problem  task, 
conditional  factors  that  affect  performance,  and  performance  standards. 
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•  The  majority  of  AAR  leaders  did  not  require  units  to  link  performance, 
corrective  actions  to  solve  problems,  nor  problem  solutions  of  any  type  to  subsequent 
training. 

Given  these  findings,  an  additional  observation  can  be  made  with  respect  to  AAR 
doctrine  outlined  in  TC  25-20.  The  doctrinal  key  points  of  an  AAR  were  not  satisfied  by 
the  majority  of  the  17  AARs.  These  points  were  reviewed  in  Chapter  I  (pp.  19  through 
22)  and,  except  for  one,  are  listed  in  Table  3.9,  Key  Points  of  an  AAR.  This  table  lists 
the  key  point  and  the  percentage  of  AARs  that  satisfied  the  point.  All  of  these  points 
concern  either  participation  or  discussion  focus.  The  point  that  weis  not  listed  is 
concerned  with  timeliness  of  the  AAR  which  could  not  be  measured. 

Table  3.9,  Key  Points  of  an  AAR 


Key  Point 

Percentage  of 
AARs  Meeting 
Criteria 

AARs  should  focus  on  the  intended  training  objectives 

100 

AARs  should  focus  on  individual  and  collective  performance 

100 

AARs  relate  to  specific  performance  standards 

30 

AARs  involve  all  participants  in  the  discussion 

24 

AAR  leaders  use  open-ended  and  leading  questions  to  prompt  and 
guide  discussion 

30 

AARs  determine  strengths  and  weaknesses 

94 

AARs  link  performance  to  subsequent  training 

12 

(DA,  TC  25-20, 1993) 
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For  the  determination  of  strengths  and  weaknesses,  an  AAR  was  considered  as 
satisfying  this  point  if  someone  identified  a  topic,  area,  or  task  as  such.  The  point  is  that 
if  strengths/weaknesses  were  required  to  be  specified  in  doctrinal  training  terms,  the 
percentage  meeting  this  criteria  would  drop  to  30%.  Furthermore,  if  the  strengths  and 
weaknesses  dictated  by  the  AAR  leader  are  not  counted,  then  the  strengths  and 
weaknesses  percentage  drops  fi'om  94%  to  53%.  The  bottom  line  for  all  of  these  points 
and  findings  is  that  AARs  are  not  being  conducted  as  participatory  discussions  focused 
on  improving  performance.  Instead,  the  majority  of  AARs  resemble  a  classroom  lecture. 
AAR  leaders  at  the  JRTC  and  NTC  are  not  implementing  doctrinal  guidance  when 
conducting  AARs.  Furthermore,  if  CTC  AARs  are  the  example  for  the  Army,  then  a 
majority  of  units,  design  engineers,  and  researchers  are  observing  ineffective  AARs  in 
terms  of  discussion  participation  and  focus. 

Notwithstanding,  the  minority  of  AARs  surveyed  -  the  4  participatory,  discussion 
focused  AARs  —  are  exemplary.  The  multiple  regression  analysis  revealed  that  the 
actions  of  these  4  AAR  leaders  are  correlated  at  a  significance  level  of  0.05.  All  4  AAR 
leaders: 

•  Emphasized  participation  and  discussion  focus  in  their  introductory  remarks; 

•  Provided  participatory  feedback  to  the  unit  to  prompt  discussion  -  they  required  unit 

members  to  talk; 

•  Used  open-ended  questions  to  prompt  discussion  and  leading  questions  to  guide 

discussion; 
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•  Used  specific  BDA  (objective  measures  of  performance)  to  focus  the  discussion;  and 

•  Focused  discussion  on  solving  a  specified  problem  in  terms  of  the  problem  task, 

conditions  that  affect  performance,  and  performance  standards;  (2  of  the  4  also 
focused  on  how  to  train  the  problem  task  in  order  to  achieve  the  desired 
performance  outcome) 

All  of  these  actions  are  in  accordance  with  or  directly  support  doctrinal  AAR  guidance. 

Discussion 

The  17  AARs  had  a  number  of  commonalties  that  could  not  be  used  in  any 
differentiating  or  regression  analysis.  Specifically,  these  were: 

1 .  The  AAR  leaders  all  followed  the  doctrinal  format.  (Table  3 .3,  Doctrinal  AAR 

Format) 

2.  None  of  the  AAR  leaders  used  a  terrain  model  nor  sand  table  during  their  AAR. 

However,  one  AAR  leader  did  use  the  actual  terrain  in  the  same  way  a  model 
would  be  used.®  He  conducted  a  terrain  walk  for  part  of  his  AAR. 

3.  OPFOR  representatives  did  not  participate  in  the  discussion  portions  of  the  AAR.  In 

fact,  all  JRTC  OPFOR  representatives,  except  one,  departed  the  AAR  before 
discussion  began.  OPFOR  representatives  did  not  attend  any  of  the  NTC  AARs. 
The  doctrinal  format  observation  is  not  surprising  given  the  emphasis  that  the  CTC 
guidance  documents  and  chain  of  command  place  on  this.  Essentially,  this  has  the  effect 

•  AAR#5. 
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of  standardizing  the  AAR  format  across  the  Army.  The  benefit  is  that  commanders  and 
units  are  all  familiar  with  the  system  and  have  common  examples  of  its  implementation. 
Hence,  this  reinforces  the  use  of  the  doctrinal  format  in  AARs  for  home-station  training. 

At  platoon  level,  the  nonuse  of  a  terrain  representation  as  an  aid  is  also 
imderstandable  in  light  of  the  short  time  period  allowed  for  AAR  preparation.  At  the 
CTCs,  the  exercise  is  free-play.  OCs  can  usually  guess  the  general  location  of  contact, 
but  not  closely  enough  to  build  a  detailed  terrain  model  of  the  area  in  time  for  the  AAR. 

It  is  even  harder  to  predict  the  exact  location  of  the  AAR  itself.  Only  if  the  contact  took 
place  in  a  relatively  small  area  (200  meter  by  200  meter  box)  is  using  the  actual  terrain  an 
effective  technique  to  aid  the  \mit’s  visualization  of  the  battlefield  and  task  performance. 

There  is  less  justification  for  not  building  a  3D  representation  of  the  terrain  at  the 
company  level.  AAR  preparation  time  is  4  hours  and  2  hours  respectively  at  the  JRTC 
and  NTC.  There  is  also  manpower  available.  The  NCOs  not  participating  in  the  platoon 
AAR  are  available  during  this  change  of  mission  period.  However,  from  personal 
experience  at  the  JRTC,  the  use  of  terrain  models  during  AARs  was  not  advocated. 

OPFOR  representatives  at  the  JRTC  were  either  platoon  leaders  or  squad  leaders 
that  had  opposed  the  unit  during  the  exercise.  All  of  these  representatives  recanted  their 
mission  and  intent  statements,  the  concept  of  operation,  and  actual  execution  of  their 
plans.  They  also  presented  observations  of  unit  performance  from  their  perspective.  The 
unit  was  then  allowed  to  question  the  OPFOR.  Of  the  10  AARs  in  which  this  occurred, 
none  of  the  units  questioned  the  OPFOR  about  their  observations.  Instead,  the  units 
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asked  trivial  questions  usually  intended  to  confirm  or  deny  the  unit’s  perceived  stereotype 
of  the  OPFOR.  Typically,  the  unit  tried  to  ascertain  what  equipment  or  information  the 
OPFOR  possessed  that  had  given  them  an  advantage.  The  longest  questioning  period 
lasted  9  minutes.  In  this  case,  the  unit  asked  questions  concerning  the  OPFOR 
disposition  and  composition.  There  was  no  discussion  of  the  unit’s  questions  beyond 
clarification  of  the  answers. 

Discussion  Participation 

Discussion  participation  is  defined  by  the  six  dependent  variables  listed  in  Table 
3.5,  Discussion  Participation  Data.  In  5  of  the  17  AARs,  100%  of  the  attendees  made  one 
or  more  declarative  statements  (comments)  during  the  AAR."  Of  these,  the  AAR  leaders 
made  concerted  efforts  to  ask  each  of  the  imit  members  a  question.  In  2  other  AARs, 
attendee  participation  was  92%. "  The  participation  rate  of  the  other  10  AARs  ranged 
from  87%  to  63%.  Of  the  7  AARs  where  the  participation  rate  was  92%  or  higher,  the 
Unit :  AAR  leader  utterance  ratio  was  above  1.00  in  4  cases.  In  other  words,  the  unit 
produced  as  many  or  more  comments  v^dth  respect  to  the  AAR  leader  in  4  AARs.  Less 
these  4,  the  comment  ratios  of  the  other  13  AARs  ranged  from  0.96  to  0.22. 

Another  aspect  of  participation  that  was  measured  was  whether  or  not  discussion 
amongst  the  unit  members,  without  the  prompting  from  the  AAR  leader’s  questions, 
occurred.  Discussion  without  the  AAR  leader  occurred  in  5  of  the  17  AARs;  3  of  these  5 

AAR#  13. 
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belong  to  the  set  of  participatory  AARs  -  AARs  with  an  attendee  participation  rate  above 
92%  and  Unit :  AAR  leader  utterance  ratio  of  1.00  or  greater.*^  The  participatory  AAR 
that  did  not  have  any  internal  discussion  amongst  the  trainees  also  had  die  only  AAR 
leader  of  the  4  that  did  not  employ  both  open-ended  and  leading  questioning  techniques. 
Of  the  5  AARs  in  which  inter-trainee  discussion  occurred,  4  of  them  had  AAR  leaders 
that  employed  both  open-ended  questions  and  leading  questions.  The  converse  of  these 
numbers  is  that  the  discussion  in  76%  of  the  AARs  (13  of  the  17)  can  be  described  as  an 
interchange  between  the  AAR  leader  and  one  attendee  at  a  time. 

A  measure  similar  to  the  Unit :  AAR  leader  utterance  ratio  is  the  Unit  comment : 
AAR  leader  question  ratio.  This  ratio  is  more  revealing  of  how  much  the  unit 
commented  beyond  replying  to  the  AAR  leader’s  questions.  For  all  of  the  AARs,  this 
ratio  value  ranged  from  4.66  to  0.67.  There  are  6  AARs  with  Unit  comment :  AAR 
leader  question  ratios  greater  than  2.00. However,  two  of  these  ratio  measures  are 
misleading.  In  the  first,  unit  comments  were  increased  by  the  AAR  leader’s  requirement 
of  unit  members  to  read  text  from  doctrinal  references.*^  In  other  words,  these  were  not 
comments  unit  members  made  of  their  own  volition.  The  second  case  involved  a  unit  and 
a  commander  with  particularly  poor  attitudes.**  The  commander  made  a  point  to 
comment  on  each  irnit  member’s  comment.  His  complaining  and  arguing  about  rules  of 

"  AAR  #3,  5,  10, 11,  and  12. 

AAR#  13  and  14. 

Discussion  amongst  the  unit  members  occurred  in  AAR  #  5, 12, 14,  and  16.  AAR  #  5, 12,  and  14  have 
high  participation  rates  and  high  Unit :  AAR  leader  utterance  ratios. 

AAR  #  5, 6, 12, 13, 14,  and  15. 

”  AAR  #6. 
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engagement  and  unfair  treatment  falsely  increased  the  comment  ratio  of  the  unit.  The 
other  4  AARs  with  Unit  comment :  AAR  leader  question  ratios  above  2.00  were  the  same 
4  AARs  that  had  Unit :  AAR  leader  utterance  ratios  above  1.00  and  participation  rates 
above  92%.'’ 

Overall,  the  majority  of  AARs  exhibited  a  low  degree  of  trainee  discussion 
participation.  A  plausible  explanation  for  this  lack  of  participation  is  Chris  Argyris’ 
theory  of  organizational  learning.  Argyris  is  the  James  B.  Conant  Professor  at  the 
Graduate  School  of  Business,  Harvard  University  and  has  completed  extensive  studies 
and  consulting  contracts  in  organizational  behavior.  On  Organizational  Learning  (1994) 
is  the  result  of  his  work  and  experience  in  this  field.  In  it,  Argyris  outlines  two  models 
that  explain  how  and  why  individuals  in  organizations  interact.  These  are  Model  I  and 
Model  II.  These  models  consist  of  a  set  of  goals  that  he  has  observed  people  to 
prevalently  use  in  a  number  of  organizational  settings.  They  are  evolutionary  in  that, 
with  guidance,  an  organization  can  evolve  from  Model  I  to  Model  II.  (Argyris,  1994) 
Although  Argyris  does  not  seem  to  have  studied  any  military  organizations.  Model  I  does 
provide  insight  into  the  individual’s  natural  resistance  to  participation  in  an  AAR. 

Argyris’  Model  I  goals  that  dominate  individual  behaviors  are  summarized  in 
Table  3.10,  Model  I  Governing  Variables.  These  governing  variables  are  the  goals  each 
individual  endeavors  to  achieve  in  daily  interactions  with  others.  To  achieve  these  goals, 
people  universally  implement  the  same  strategies.  These  are  to:  1)  “advocate  your 
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Table  3.10,  Model  I  Governing  Variables 
Individual’s  Goal 

1 .  Strive  for  unilateral  control. 

2.  Minimize  losing  and  maximize  winning. 

3.  Minimize  the  expression  of  negative  feelings. 

4.  Be  rational  -  do  not  do  anything  illogical. 

(Argyris,  1994, 26  and  150  to  151) 

views  without  encouraging  inquiry”  in  order  to  obtain  unilateral  control  of  the  interaction 
and  “win”  your  point,  and  2)  “unilaterally  save  face  -  your  own  and  [if  possible]  other 
people’s”  in  order  to  minimize  upsetting  others’  pursuit  of  the  same  goals  or  “making 
them  defensive.”  Each  individual  learns  these  goals  and  strategies  socially  throughout  his 
formative  years.  (Argyris,  1994, 26) 

Obviously,  people  acting  out  Model  I  behaviors  will  be  resistant  to  any  situation 
or  person  that  prevents  them  from  doing  so.  Thus,  in  AARs  the  hesitance  or  resistance  to 
participation  in  a  discussion  can  be  accounted  for  by  Model  I.  Each  individual  naturally 
wants  to  avoid  embarrassment,  first  of  himself  and  second  of  others.  Furthermore,  if  one 
cannot  advocate  his  opinion  without  being  questioned  about  it,  he  is  less  likely  to 
volxmteer  the  opinion  in  the  first  place.  This  environment  is  typical  of  an  AAR.  As 
Argyris  found  in  so  many  of  his  case  studies,  individuals  who  are  placed  in  these  types  of 

”  AAR  #5, 12, 13,  and  14. 
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environments  resist  saying  anything  at  first,  then  become  argumentative  and  employ 
defensive  reasoning  strategies  when  forced  to  participate.  (Argyris,  1994) 

Given  that  a  high  rate  of  participation,  large  comment  ratio,  and  discussion 
amongst  the  unit  are  all  desirable,  the  4  AARs  that  meet  this  criteria  require  some 
scrutiny.  In  all  but  1,  the  AAR  leader  employed  both  focused,  open-ended  and  leading 
questions.  In  the  exception,  the  AAR  leader  only  used  focused,  open-ended  questions.'* 
This  exception  is  also  the  only  one  of  the  group  in  which  discussion  'without  the  AAR 
leader  did  not  occur.  In  all  17  AARs,  5  AAR  leaders  used  both  focused,  open-ended  and 
leading  questions;  3  AAR  leaders  employed  one  or  the  other  type  questions;  and  9  AAR 
leaders  used  neither. 

Additionally,  the  4  leaders  of  the  participatory  AARs  were  the  only  ones  to  use 
specific,  non-qualitative  BDA  during  the  discussion  portions  of  the  AAR.  Terrain  and 
contact  sketches  were  also  used  to  aid  discussion  in  3  of  the  4  AARs.  The  AAR  leader 
who  did  not  use  sketches  used  the  actual  terrain  instead.*’  He  walked  the  unit  along  the 
objective  so  that  unit  members  could  explain  what  happened,  where  it  happened,  and 
why.  Hence,  the  terrain  walk  substituted  for  the  sketches.  A  total  of  1 5  AAR  leaders 
referenced  large  maps  and  terrain  and  contact  sketches  at  some  point  in  the  discussion. 

Other  characteristics  of  these  4  AARs  were  unit  attitude  and  unit  leader 
facilitation  actions.  Half  of  the  4  were  rated  as  neutral  (qualitative  code  of  “0”)  for  both 


In  AAR  #  13,  the  AAR  leader  did  not  use  leading  questions. 
AAR  #5. 
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vinit  attitude  and  leader  helpfulness,  and  the  other  2  AARs  were  rated  as  good  (qualitative 
code  of  “1”)  for  both  factors.^  Of  all  17  AARs; 

•  3  units  had  a  poor  attitude  and  an  unit  leader  who  made  negative  comments 

(qualitative  code  of  “-1”);^’ 

•  3  units  had  a  poor  attitude  and  an  unit  leader  who  was  neutral,  but  did  not  correct  the 

unit’s  attitude;^ 

•  2  units  had  a  neutral  attitude  and  a  leader  who  made  negative  comments;^^ 

•  7  units  had  a  neutral  attitude  and  a  neutral  leader;^"*  and 

•  2  units  had  a  positive  attitude  and  a  leader  who  proactively  aided  the  AAR  process.^ 
Unfortunately,  8  of  the  17  AARs  had  a  imit  with  a  poor  attitude,  a  leader  with  a  poor 
attitude,  or  both.  Thus,  half  of  the  units  displayed  tendencies  in  keeping  with  Argyris’ 
Model  I  behaviors.  (Argyris,  1994)  Experience  in  and  with  combat  arms  units  suggests 
those  that  habitually  assess  themselves  against  established  performance  standards  are  not 
argumentative,  defensive,  nor  hostile.  These  units  look  for  solutions  rather  than  excuses. 

AAR  Leader  and  Unit  Actions  Affecting  Discussion  Participation 


AAR  #  5  and  13  received  scores  of  “0”  and  AAR  #  12  and  14  received  scores  of  “1.” 
AAR#2, 15,andl7. 

“  AAR  #3, 7,  and  10. 

“  AAR  #8  and  16. 

AAR#  1,3,  5, 6, 9, 11,  and  13. 

“  AAR  #12  and  14. 
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Regression  analysis  confirmed  a  number  of  the  trends  found  in  die  previous 
analysis.  The  models  are  summarized  in  Tables  3.1 1,  Participation  Factors,  and  3.12, 
Discussion  Focus  Factors.  The  dependent  variable  and  five  independent  variables  that 
had  the  largest  effect  are  listed  for  each  model.  Each  effect’s  relative  order  of  magnitude 
is  given  in  parenthesis.  This  order  of  magnitude  is  only  relative  to  the  ofiier  effects 
within  the  equation.  This  number  is  not  comparable  between  models.  The  coefficient  of 
determination  (R^)  for  each  model  is  listed  in  brackets  beneath  each  dependent  variable 
(leftmost  column  of  the  table). 

The  first  dependent  variable  analyzed  was  the  Unit :  AAR  leader  utterance  ratio. 
(Appendix  B,  Output  B-2)  At  a  statistical  significance  level  of  0.05  and  a  R^  =  90.3%, 
the  factors  that  had  the  largest  effect  on  this  variable  were:  the  AAR  leader  giving  an 
introduction  that  emphasized  participation,  the  AAR  leader’s  use  of  open-ended  and 
leading  questions,  and  the  AAR  leader  providing  participatory  feedback  to  the  unit 
combined  with  the  use  of  sketches/large  maps  during  discussion.  Respectively,  an 
introduction  emphasizing  participation  and  the  interaction  of  open-ended  and  leading 
questions  affected  the  utterance  ratio  3.6  and  3.0  times  more  than  the  independent 
variable  with  the  smallest  effect  (the  use  of  specific  BDA).  The  interaction  of 
participatory  feedback  and  the  use  of  sketches/enlarged  maps  was  1 .9  times  the  lowest 
order  of  magnitude  in  effect.  These  are  actions  that  were  present  in  the  AARs  with  a 
higher  Unit :  AAR  leader  utterance  ratio. 
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Table  3.11,  Discussion  Participation  Factors 


Dependent 

Variable 

Independent 
variable  w/the 
largest  effect 

Independent 
variable  w/2"‘* 
largest  effect 

Independent 
variable  w/S"** 
largest  effect 

Independent 
variable  w/4^ 
largest  effect 

Independent 
variable  w/5^ 
largest  effect 

Unit :  AAR 

Introduction 

Interaction: 

Number 

Interaction: 

Unit  specified 

leader 

establishing  a 

used  leading  & 

questions  asked 

Participatory 

Solutions 

utterance 

participatory 

open-ended 

by  AAR  leader 

feedback 

planned'^^ 

ratio 

climate 

questions 

(-2.0) 

provided  to 

(-1.7) 

[R"=90.3%] 

(3.6) 

(3.0) 

Unit  &  used 

sketches/maps 

(1.9) 

%Unit 

Used  sketches 

Interaction: 

AAR  exceeded 

Participation 

&  maps  in 

Used  leading  & 

specified  time 

in  the 

discussion 

open-ended 

limit 

Discussion 

(1.5) 

questions 

(-1) 

[R"=52.7%] 

(1.3) 

Unit 

Interaction: 

Used  open- 

Number 

Unit  specified 

Introduction 

comment : 

Participatory 

ended 

questions  asked 

Solutions 

establishing  a 

AARldr 

feedback 

questions 

by  AAR  leader 

planned 

participatory 

question  ratio 

provided  to 

(1.5) 

(-1.2) 

(-1.1) 

climate 

[R2=95.4%] 

Unit  &  used 

(1.1) 

specific  BDA 

(3.5) 

#  Questions 

Participatory 

Unit  leader 

Interaction:  used 

asked  by  the 

feedback 

aided 

open-end 

unit 

provided  to 

discussion 

questions  &  Intro 

[R"=96.7%] 

Unit 

(1.0) 

establishing  a 

(2.4) 

participatoiy 

climate 

(1.0) 

Discussion 

Unit  specified 

Interaction: 

Interaction:  used 

Introduction 

amongst  unit 

Solutions 

used  open-end 

specific  BDA  & 

establishing  a 

members 

planned 

questions  & 

sketches/maps 

participatory 

[R2=69.8%] 

(5.0) 

specific  BDA 

(-4.0) 

climate 

(4.0) 

(-1.0) 

(Appendix  B,  Regression  Analysis,  Outputs  B-2,  B-3,  B-4,  B-5,  and  B-6) 


This  variable  is  a  qualitative  condition,  either  the  unit  specified  solutions  planned  or  they  did  not.  These 
exclude  those  specified  by  the  AAR  leader. 
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The  number  of  questions  asked  by  the  AAR  leader  and  the  unit  action  of 
discussing  solutions  planned  had  a  negative  effect  on  the  unit  speaking  more  than  the 
AAR  leader.  The  presence  of  variables  with  a  negative  (-)  effect  indicates  that  the 
majority  of  AARs  did  not  perform  those  actions  (regressor  variables).  Thus,  a  greater 
number  of  AAR  leader  questions  did  not  lead  to  the  trainees  speaking  more  (relative  to 
the  AAR  leader);  and  the  discussion  of  solutions  planned  by  the  unit  was  not  done  in  the 
majority  of  AARs. 

The  second  dependent  variable  concerned  how  many  of  the  unit  members 
participated  in  the  discussion.  (Appendix  B,  Output  B-3)  However,  the  regression 
analysis  was  inconclusive  because  only  57.2%  of  the  data’s  variance  could  be  explained 
by  the  model  (R^  =  57.2%).  This  is  not  surprising  since  most  of  the  AAR  leaders  were 
observed  to  specifically  ask  a  question  of  each  unit  member.  In  fact,  the  most  common 
AAR  leader  questions  of  a  unit  member  concerned  the  assessment  of  something  another 
unit  member  had  stated  or  what  that  unit  member  had  done  at  a  specific  point  in  the 
exercise.  Many  of  these  questions  were  of  the  form:  is  that  right.  Sergeant?;  what  do  you 
think  of  that.  Specialist?;  or  what  were  you  doing  at  that  time?;  and  what  happened  over 
there.  Private?  Hence,  the  participation  rate  does  not  strongly  correspond  with  any  of  the 
AAR  leader  or  unit  actions  measured. 

The  Unit  comment :  AAR  leader  question  ratio  was  also  analyzed  as  a  dependent 
variable.  (Appendix  B,  Output  B-4)  At  a  R^  of  95.4%,  the  interaction  between  the  AAR 
leader  providing  participatory  feedback  to  the  unit  and  using  specific  BDA  during  the 
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discussion  had  the  largest  effect  on  the  unit  submitting  more  comments  relative  to  the 
number  of  AAR  leader  questions.  The  AAR  leader’s  use  of  open-ended  questions  and  an 
AAR  introduction  emphasizing  a  participatory  climate  also  correlated  with  a  higher  Unit 
comment  to  AAR  leader  question  ratio.  The  AAR  leader  emphasizing  participation  in  the 
introduction,  giving  participatory  feedback  to  the  unit,  and  using  open-ended  questions 
were  actions  that  were  also  present  in  the  Unit :  AAR  leader  utterance  ratio  model.  This 
reinforces  the  suggestion  that  these  AAR  leader  actions  are  the  keys  to  unit  members 
saying  more  and  the  AAR  leader  saying  less.  Likewise,  the  fact  that  the  same  actions, 
number  of  AAR  leader  questions  and  discussion  of  solutions  planned,  also  showed  up  as 
negative  effects  again  reinforces  the  point  that  they  were  not  performed  in  the  majority  of 
AARs.  The  total  number  of  questions  asked  by  the  AAR  leader  varied  inversely  with  the 
ratio.  Thus,  in  the  AARs  with  a  low  comment  to  question  ratio,  AAR  leaders  still  asked  a 
large  number  of  questions  and  the  units  did  not  discuss  the  subject  of  solutions  planned. 

The  number  of  questions  asked  by  vmit  members  seemed  to  be  positively 
influenced  by  the  AAR  leader’s  participatory  feedback,  the  unit  leader’s  help  in 
facilitating  the  AAR,  and  the  interaction  of  the  AAR  leader  emphasizing  participation  in 
the  introduction  and  then  employing  open-ended  questions.  (Appendix  B,  Output  B-5) 
Participatory  feedback  had  roughly  twice  the  effect  of  the  other  two  variables  during  the 
discussion.  It  should  be  noted  that  the  variables  for  the  unit  leader  aiding  discussion  and 
the  unit  having  a  positive  attitude  were  highly  correlated.  This  confounded  the  analysis 
and  the  two  variables  had  to  be  combined.  Since  the  unit  leader  is  responsible  for  and  in 
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the  most  influential  position  to  affect  the  unit’s  attitude,  the  combined  variables  were 
defined  qualitatively  as  “unit  leader  aided  discussion”  and  “unit  leader  hindered 
discussion.” 

The  last  participation  variable  analyzed  was  discussion  amongst  unit  members. 
(Appendix  B,  Output  B-6)  This  was  a  dialogue  between  two  or  more  unit  members 
without  the  AAR  leader  intervening.  At  a  =  69.8%,  this  discussion  corresponded  with 
the  unit  specifying  solutions  planned  and  the  interaction  between  the  AAR  leader’s  use  of 
open-ended  questions  and  specific  BDA.  The  AAR  leaders’  introduction  emphasizing 
participation  and  the  interaction  of  specific  BDA  and  sketch/map  use  negatively  affected 
the  model.  This  confirms  that  the  majority  of  AARs  did  not  have  discussion  amongst  die 
participants  occur  spontaneously  but  still  may  have  performed  one  or  both  of  the 
independent  variables:  gave  a  participatory  introduction  and  used  open-ended  questions 
in  combination  with  specific  BDA. 

The  models  of  Unit :  AAR  leader  utterance  ratio.  Unit  comment :  AAR  leader 
question  ratio,  and  number  of  questions  asked  by  the  unit  all  had  (high)  R^  values  greater 
than  90%.  On  die  other  hand,  the  model  describing  the  discussion  amongst  the  unit  was 
moderately  adequate  (R^  =  69.8%)  and  the  one  for  percent  participation  did  not  describe 
the  data  well.  Notwithstanding,  none  of  these  models  contradict  the  trend  analysis  of  the 
4  participatory  AARs.  They  do  suggest  some  support  for  emphasizing  participation  in 
the  AAR  introduction,  continuously  providing  participatory  feedback  to  the  unit,  and 
guiding/focusing  the  discussion  with  open-ended  and  leading  questions  in  order  to 
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increase  trainee  discussion  participation.  The  use  of  specific  BDA  and  sketches/enlarged 
maps  also  were  significant  effects  on  participation. 

The  use  of  open-ended  and  leading  questions  both  played  roles  in  increasing 
participation.  The  use  of  open-ended  questions  varied  directly  with  all  the  dependent 
variables  except  the  percentage  of  the  unit  commenting  during  the  discussion.  This  is 
most  likely  a  result  of  following  doctrine  and  the  emphasis  on  participatory  discussion. 
AAR  doctrine  states  that  the  use  of  open-ended  and  leading  questions  is  the 
recommended  technique  for  the  AAR  leader  to  conduct  the  AAR.  (DA,  TC  25-20, 4-3) 

There  seems  to  be  little  doubt  that  the  AAR  leader’s  skill  in  engaging  the  trainees 
in  the  discussion,  use  of  various  aids,  and  participatory  maintenance  of  the  discussion  all 
affect  the  level  of  trainee  participation  observed.  Besides  Argyris’  theory  (1994),  the 
unit’s  previous  AAR  experienee  may  be  another  plausible  explanation  for  their  low  level 
of  participation.  The  majority  of  unit’s  may  never  have  experienced  a  participatory  AAR. 
If  feedback  is  always  provided  via  lecture-critique,  they  would  not  know  how  to 
participate  in  a  discussion  of  performance. 

Discussion  Foeus 

Solutions  were  developed  in  all  but  1  (one)  AAR.^’  For  these  16  AARs,  a  total  of 
36  solutions  developed  were  adequately  articulated.  However,  only  3  of  them  adequately 
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described  solutions  planned?^  Of  the  36  solutions  developed,  a  total  of  4  were  resolved 
into  solutions  planned.  Hence,  the  AAR  leader  or  unit  described  feasible  training  tasks, 
conditions,  and  performance  standards  to  correct  identified  problems  in  18%  of  the 
AARs. 

If  the  munber  of  solutions  planned  and  solutions  developed  by  the  unit  are  only 
counted,  then  the  numbers  are  4  and  17  respectively.  In  other  words,  removing  die  AAR 
leaders’  contributions  attributes  less  then  half  of  the  solutions  developed  and  all  of  the 
solutions  planned  to  the  unit.  Of  the  4  solutions  planned,  3  can  be  attributed  to  the  AARs 
with  high  participation  rates  and  comment  ratios.^’  Of  the  17  solutions  developedhy 
units,  9,  or  69%,  were  developed  in  these  high  participation  AARs.^° 

In  the  AARs  in  which  solutions  were  developed  and  planned,  the  AAR  leaders  all 
continually  emphasized  that  the  solution  was  the  objective  of  the  discussion.  In  other 
words,  each  AAR  leader  guided  the  discussion  to  a  specific  end  -  a  solution  to  the 
specified  problem.  Whereas  the  AAR  leaders  who  did  not  achieve  solutions  in  their 
AARs  did  not  emphasize  this  objective  continuously.  In  12  of  16  AARs,  the  AAR  leader 
stated  that  solutions  to  problems  were  the  objective  of  the  AAR  in  his  introductory 
remarks.^*  This  objective  was  continuously  emphasized  in  only  3  AARs;  the  3  AARs  in 
which  solutions  planned  were  specified. 


“  AAR  #3, 12,  and  14. 

AAR#  12 and  14. 

AAR  #5, 12, 13,  and  14. 

The  introduction  of  AAR  #16  was  not  filmed. 
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The  lengths  of  the  discussion  periods  were  recorded  for  each  AAR  but  were 
inconclusive.  The  discussion  times  were  measured  as  a  percent  of  the  total  AAR  times. 
The  values  ranged  from  92%  to  24%. 

AAR  Leader  and  Unit  Actions  Affecting  Discussion  Focus 

The  regression  models  for  the  number  of  unit  specified  solutions  developed  and 
planned,  and  the  total  number  of  solutions  developed  and  planned  are  summarized  in 
Table  3.12,  Discussion  Focus  Factors.  The  level  of  statistical  significance  for  these 
models  is  an  Alpha  (a)of  0.05.  Again,  the  number  in  parenthesis  listed  beneath  each 
factor  is  the  effect’s  order  of  magnitude  relative  to  the  other  factors  in  the  regression 
equation.  Also,  the  models’  R^  values  are  bracketed  beneath  the  dependent  variable. 

Reviewing  Table  3.12,  Discussion  Focus  Factors,  it  is  obvious  that  the  models  of 
Unit  and  Total  solutions  planned  are  the  same.  At  the  same  time,  the  models  of  Unit  and 
Total  solutions  developed  seem  completely  different.  Since  a  unit  first  develops  a 
solution  before  planning  its  implementation,  the  models  of  unit  and  total  solutions 
developed  will  be  investigated  first. 

In  9  of  17  AARs,  the  units  specified  solutions  developed.  The  use  of  open-ended 
questions  and  the  unit  leader  aiding  discussion  had  the  highest  correlation  with  this 
dependent  variable.  The  primary  reason  for  this  correlation  is  that  these  9  AARs 
accounted  for  100%  of  all  the  observations  in  which  open-ended  questions  were 
employed  and  the  unit  leader  helped. 
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Table  3.12,  Discussion  Focus  Factors 


Dependent 

Variable 

Independent 
variable  w/ 
largest  effect 

Independent 
variable  w/2"‘^ 
largest  effect 

Independent 
variable  w/S"** 
largest  effect 

Independent 
variable  w/4* 
largest  effect 

Independent 
variable  w/5^ 
largest  effect 

#Unit 

solutions 

developed 

[R2=87.5%] 

Used  open- 
ended 
questions 
(1.8) 

Unit  leader 
aided 
discussion 
(1.6) 

Unit  leader 
hindered 
discussion 
(-1.4) 

Used  sketches 
&  maps  in 
discussion 
(-1.2) 

AAR  exceeded 
specified  time 
limit 
(1.1) 

#  Total 
solutions 
developed 
[RM3.5%] 

Participatory 
feedback 
provided  to 
Unit 
(1.5) 

Used  specific 
BDA 
(1.0) 

#Unit 

solutions 

planned 

[R^99.7%] 

Unit  leader 
aided 
discussion 
(30) 

Interaction: 
Participatory 
feedback 
provided  to 
Unit  &used 
specific  BDA 
(-29) 

#  questions 
asked  by  unit 
(11.0) 

Used  specific 
BDA 
(-2.0) 

Use  of  leading 
questions 
(1.0) 

#  Total 
solutions 
planned 
[R^99.7%] 

Unit  leader 
aided 
discussion 
(30) 

Interaction: 
Participatory 
feedback 
provided  to 
Unit  &used 
specific  BDA 
(-29) 

#  questions 
asked  by  unit 
(11.0) 

Used  specific 
BDA 
(-2.0) 

(Appendix  B,  Regression  Analysis,  Outputs  B-7,  B-8,  B-9,  and  B-10) 


The  primary  reason  for  the  difference  between  the  Unit  and  Total  solutions 
developed  models  is  the  number  of  observations  each  dependent  variable  is  based  upon. 
While  units  specified  solutions  developed  in  9  AARs,  Total  solutions  developed  were 
specified  (by  the  AAR  leader  or  the  unit)  in  16  of  the  17  AARs.  The  model  based  upon  a 
larger  number  of  observations  (16)  had  to  account  for  a  larger  variability  in  AAR  leader 
and  unit  actions.  Many  of  the  AAR  leaders  in  those  7  additional  AARs  did  not  perform 
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the  same  actions  as  those  leaders  who  led  the  9  AARs  with  Unit  specified  solutions 
developed.  This  resulted  in  a  low  (43.5%). 

The  AAR  leader  actions  that  coincided  with  the  Total  solutions  developed  in  the 
AAR  (by  either  the  AAR  leader  or  the  unit)  were  the  provision  of  participatory  feedback 
to  the  unit  and  the  use  of  specific  BDA  during  the  discussion.  Thus,  the  AAR  leaders 
guided  the  AAR  discussion  with  objective  performance  outcome  data  while  continuously 
providing  participation  feedback  to  the  unit.  In  other  words,  they  emphasized  unit 
discussion  participation  more  than  the  other  AAR  leaders.  This  participation  feedback 
went  hand  in  hand  with  discussion  focus  feedback  -  explicit  statements  reminding  the 
unit  that  the  objective  of  the  discussion  was  to  solve  the  problem  at  hand. 

The  models  for  Unit  specified  solutions  planned  and  (Total)  both  AAR  leader  and 
unit  specified  solutions  planned  are  the  same  because  each  is  based  upon  the  same  3 
AARs.  (Table  3.12,  Discussion  Focus  Factors,  and  Outputs  B-9  and  B-10)  There  were 
only  3  AARs  in  which  solutions  were  planned;  all  of  these  were  specified  by  the  unit 
during  the  discussion.  The  unit  leader  aiding  discussion  during  the  AAR  had  a  very  large 
effect  on  the  number  of  solutions  planned  (+30).  Again,  it  should  be  noted  that  this 
variable  and  the  attitude  of  the  unit  had  to  be  combined  because  of  multicollinearity 
problems.  On  the  other  hand,  the  interaction  variable  of  participatory  feedback  and  the 
use  of  specific  BDA  had  a  strong  negative  effect  (-29).  This  negative  effect  highlights 
the  fact  that  the  majority  of  AAR  leaders  did  not  perform  these  actions  together.  The 
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large  order  of  magnitudes  of  the  coefficients  for  these  variables  indicates  that  both  of 
these  interactions  were  important  to  developing  solutions. 

The  number  of  questions  asked  by  the  unit  also  varied  directly  with  the  number  of 
solutions  planned.  This  makes  sense  in  terms  of  the  objectives  of  the  AAR  -  to  solve 
performance  problems  in  order  to  improve  performance.  If  this  objective  is  reiterated 
during  the  AAR,  then  trainees  will  focus  on  it  and  direct  their  questions  towards  its 
solution/goal.  An  observation  noticed  but  not  measured  during  data  collection,  was  that 
the  unit  seemed  to  participate  in  the  discussion  more  when  a  problem  was  near  solution 
(as  opposed  to  the  problem  definition  phase). 

The  most  prevalent  characteristic  of  these  models  seems  to  be  the  presence  of 
participatory  feedback  in  each.  This  is  most  likely  the  result  of  the  participatory  AARs 
accounting  for  a  large  portion  of  the  solutions  developed  and  planned.  Therefore,  the 
most  significant  effects  are  those  AAR  leader  actions  that  facilitate  discussion  focus  -  use 
of  specific  BDA,  sketches  and  maps,  open-ended  and  leading  questions. 

Time 

The  total  time  length  of  each  AAR  was  measured  in  minutes  and  then  recorded 
qualitatively.  A  total  of  7  AARs  exceeded  their  CTC  time  limits.  Of  the  4  high 
participation  AARs,  2  exceeded  the  time  limits  by,  respectively,  21  minutes  and  9 
minutes.^^  Respectively,  these  AAR  leaders  spent  90%  and  86%  of  the  total  AAR  time 

AAR# 5 and  12. 
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on  discussion.  Both  of  the  units  and  leaders  in  each  AAR  were  observed  to  have  either  a 
positive  or  neutral  attitude  suggesting  that,  for  these  cases,  time  had  no  bearing  on  the 
attitude  of  the  unit  or  leader.  Participants  in  both  AARs  were  visibly  fatigued  and, 
although  this  was  not  measured,  it  was  apparent  to  the  observers  that  participatory 
comments  were  less  frequent  and  unit  member  attention  was  waning. 

AAR  Leader  and  Unit  Actions  Affecting  AAR  Length  (Time) 

In  the  regression  analysis  of  a  =  0.05,  the  percent  of  total  AAR  time  spent  on 
discussion  could  not  be  modeled  with  AAR  leader  actions,  umt  actions,  or  their 
interactions.  (Appendix  B,  Regression  Analysis,  Output  B-1 1)  The  percent  of  total  AAR 
time  spent  on  discussion  ranged  from  24%  to  92%.  The  percentages  for  the  4 
participatory,  discussion  focused  AARs  were  52%,  71%,  86%,  and  90%.  These 
percentages  exemplify  the  variance  of  this  variable  in  all  1 7  AARs.  Although  it  could  not 
be  confirmed  statistically,  it  seems  that  a  discussion  by  a  number  of  people  should  take 
longer  than  a  discussion  between  two  people.  Therefore,  more  time  should  be  needed  for 
the  articulation  and  consideration  of  multiple  viewpoints. 


Conclusion 

Presuming  these  17  cases  are  representative  of  the  CTCs,  then  the  findings  of  this 
survey  suggest  that  the  majority  of  AARs  conducted  at  platoon  and  company  level  maybe 
ineffective  with  respect  to  discussion  participation  and  focus.  If  this  is  true,  speculation 
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of  the  implications  is  not  pleasant.  Most  significantly,  the  AARs  that  the  combat  arms 
units  and  research  and  development  communities  use  as  models  are  fundamentally 
flawed.  In  particular,  R&D  has  the  potential  to  design  AAR  support  systems  that  aid 
non-participatory  and  unfocused  AARs. 

A  minority  of  the  AARs  surveyed  could  be  characterized  as  both  participatory  and 
topic  focused.  In  these,  flie  AAR  leaders’  actions  that  affected  discussion  participation 
and  focus  were  found  to  be  correlated  and  interdependent.  It  is  apparent  that  the 
synthesis  of  these  actions  produces  the  desired  trainee  participatory  discussion  of 
performance  problems  and  solutions.  All  of  these  actions  are  specified  in  doctrinal  and/or 
CTC  guidance  and  can  be  trained  in  a  performance  oriented  manner. 

For  these  17  AARs,  the  use  of  aids  to  facilitate  discussion  participation  and  focus 
is  limited  to  specific  BDA,  the  doctrinal  performance  standards  listed  in  the  MTPs,  and 
enlarged  sketches  and  maps.  In  2  of  the  participatory  AARs,  the  AAR  leaders  had  the 
unit  draw  their  own  sketches  to  explain  their  statements.  In  the  third  participatory  AAR, 
the  actual  terrain  was  used  as  a  discussion  medium.  The  last  participatory  AAR  leader 
used  his  own  sketches  in  conjunction  with  the  MTP  standards  to  facilitate  the  discussion 
and  problem  solving.  The  point  here  is  that  they  all  used  some  type  of  aid  to  facilitate 
discussion  of  the  problem  and  solution. 

Beyond  the  AAR  leader’s  actions,  the  unit’s  command  climate  and  previous  AAR 
experience  may  affect  discussion  participation  and  focus.  Additionally,  OPFOR 
participation  and  use  of  a  3D  terrain  model  during  the  discussion  could  not  be  measured. 
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Both  of  these  variables  were  cited  in  the  AAR  research  literature  as  important  to 
discussion  participation  and  focus.  To  be  conclusive,  future  research  must  account  for 
these  deficiencies. 
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CHAPTER  IV 


PROPOSED  SOLUTION 


"Three-fourths  of  those  tilings  upon  which  action  in  War  must  be 
calculated,  are  hidden  more  or  less  in  the  clouds  of  great  xmcertainty. 
Here,  then,  above  all,  a  fine  and  penetrating  mind  is  called  for,  to  search 
out  tile  truth  by  the  tact  of  its  judgment. 

An  average  intellect  may,  at  one  time,  perhaps  hit  upon  this  truth 
by  accident;  and  extraordinary  courage,  at  another,  may  compensate  for 
the  want  of  this  tact;  but  in  the  majority  of  cases  the  average  result  will 
always  bring  to  light  the  deficient  imderstanding." 

—Carl  von  Clausewitz 
On  War.  1830 


Review  of  the  Problem 

The  purpose  of  the  AAR  is  to  improve  individual  and  collective  trainee 
performance.  (DA,  FM  25-101, 1990,  G-1)  Trainees  can  only  improve  performance  if 
they  leam:  what  was  performed  correctly/incorrectly,  why  they  performed  in  that 
manner,  and  how  to  sustain/correct  their  performance.  Thus,  an  effective  AAR  is  one  in 
which  performance  problems  are  identified,  defined,  and  solved  in  such  a  manner  that  the 
trainees  leam  the  solution  -  they  are  able  to  “adopt  a  course  of  action  to  correct 
problems.”  (DA,  TC  25-20, 1993, 4-4)  Furthermore,  the  examination  of  doctrine  and 
review  of  AAR  literature  reveals  that  the  AAR  has  four  elements  of  effectiveness:  1) 
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discussion  participation  —  trainee  participation  in  the  problem  solving  discussion;  2) 
discussion  focus  -  focusing  the  discussion  on  a  single  problem  and  solution;  3)  learning 
reinforcement  -  mental  and  physical  practice  of  the  solution;  and  4)  time  -  timeliness  of 
the  AAR  after  training  and  timeliness  of  the  learning  reinforcement.  The  first  two 
elements  derive  directly  from  AAR  doctrine  as  shown  by  the  doctrinal  definition.  The 
AAR  is: 

a  method  of  providing  feedback  to  units  by  involving  participants  in  the 
training  diagnostic  process  in  order  to  increase  and  reinforce  learning.  The 
AAR  leader  guides  participants  in  identifying  deficiencies  and  seeking 
solutions.  (DA,  FM  25-101,  1990) 

Learning  reinforcement  and  timeliness  considerations  derive  from  cognitive  learning 
theory.  (Chapter  II) 

Given  this  description  of  the  AAR,  the  current  training  simulation  AAR  systems 
were  investigated  and  CTC  AARs  studied.  The  result  of  this  work  suggests  the  following 
multi-faceted  problem:  the  majority  of  small  unit  training  system  AARs  may  be 
ineffective.  The  majority  of  the  AARs  are  not  problem  solving  sessions  nor  are  AAR 
leaders  following  doctrinal  AAR  guidance  with  respect  to  discussion  participation. 
Ineffective  AARs  are  occurring  at  the  CTCs  and  since  they  are  the  model  for  the  training 
and  R&D  communities,  this  ineffectiveness  may  be  propagated. 

Currently,  AAR  systems  and  R&D  efforts  are  focused  on  making  the  preparation 
of  the  AAR  more  efficient.  The  primary  focus  of  ST  AARS  is  not  the  improvement  of 
unit  performance,  but  the  collection  and  networking  of  standardized  data  elements  from 
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live,  virtual,  and  constructive  training  exercises.  Although  the  second  draft  is  incomplete, 
presentation  formats  for  this  data  are  also  standardized  by  STAARS.  (BCTD,  1995)  The 
result  of  these  efforts  has  been  the  standardization  of  the  presentation  medium  for  the 
data. 

STAARS,  UPAS,  CCTT,  ATAFS,  and  STRIPES  are  committed  to  a  single 
presentation  method.  The  products  required  by  STAARS  and  those  produced  by  CCTT, 
ATAFS,  and  STRIPES  are  specifically  formatted  for  one  type  of  media  presentation  -  the 
television  screen  or  high  resolution  monitor  with  sound  system.  To  date,  AAR  system 
development  has  focused  on  the  efficient  collection  of  performance  data  and  the 
standardized  display  of  that  data  -  in  format  and  medium.  This  is  just  one  of  the  system 
components  as  shown  in  Figures  3.1,  Conceptual  CTC  AAR  System  and  3.2,  CTC  AAR 
Components  of  Chapter  III. 

A  major  advantage  of  simulation  training  is  that  a  unit  can  execute  multiple 
simulations  of  an  exercise  in  a  shorter  time  period  and  at  lower  cost  than  live  training 
allows.  Consequently,  the  unit  will  conduct  multiple  AARs  —  one  for  each  training 
iteration.  Combine  this  repetition  with  a  single  presentation  medium,  mostly  displaying 
statistical  charts  and  graphs,  and  the  product  is  trainee  boredom.  The  trainer  must  be  able 
to  highlight  the  important  AAR  points  in  a  manner  that  catches  the  trainees’  attention. 

Even  with  different  AAR  points,  a  repeated  presentation  method  will  cause  the 
trainee  to  learn  the  method,  not  the  presentation  content.  For  example,  being  shown  five 
statistical  reports  in  the  STAARS  format  will  teach  a  person  to  differentiate  a  statistical 
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report  from  any  other  type  format.  In  other  words,  he  can  recognize  that  format.  Unless 
the  content  of  each  statistical  report  was  signified,  he  probably  will  not  remember  the  five 
points  that  were  communicated  by  the  statistics.  After  four  AARs  in  one  day  of  training, 
trainees  will  be  able  to  quickly  identify  the  format  for  a  battle  summary,  battle  set,  sketch, 
statistical  report,  and  word  slide  but  will  be  less  likely  to  remember  what  the  learning 
points  of  each  were.  This  is  simple  pattern  recognition  and  the  productions  that  represent 
the  patterns  are  being  continually  reinforced  in  this  situation. 

The  above  conclusion  is  not  as  narrow  in  consideration  as  it  seems.  The  majority 
of  the  CTC  AARs  surveyed  are  not  participatory  discussions  involving  the  trainees. 
Additionally,  the  majority  of  AAR  leaders  did  not  allow  nor  force  the  trainees  to  solve 
ftieir  own  performance  problems.  This  trainee  participation  in  problem  solution  is  the 
signifying  factor  of  the  standardized  information  formats  and  did  not  occur  in  76.5%  of 
AARs  observed.  Thus,  after  being  lectured  to  for  an  hour,  four  times  in  a  day,  the 
trainees  will  have  learned  the  information  format  and  perhaps  written  down  some  of  the 
AAR  leader’s  points.  But  few,  if  any,  of  the  trainees  will  have  articulated  the  cause  and 
effect  factors  and  elaborated  their  relationships  in  terms  of  what  they  already  know. 

A  second  observation  of  relying  on  a  single  presentation  medium  is  that  it  does 
not  prompt  audience  participation;  the  content  signifying  effect.  The  feedback  on  the 
screen  is  perceived  as  the  trainer’s,  not  the  trainees.’  This  is  especially  true  when  the 
presentation  medium  is  controlled  by  the  AAR  leader  to  support  the  AAR  leader’s  point. 
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Whether  the  AAR  leader  or  the  AAR  system  produces  the  presentation,  the  trainee  still 
does  not  perceive  the  input  as  his  own. 

The  trainee  needs  the  ability  to  input  into  the  presentation  to  make  the  feedback 
“his  own.”  Trainees  are  much  more  likely  to  correct  a  problem  for  which  they  have 
identified  and  developed  a  solution  for.  With  the  current  AAR  systems,  the  trainer  is  left 
to  soliciting  discussion  from  the  audience  with  verbal  statements  and  the  pre-designated 
AAR  products. 

To  stimulate  audience  participation,  the  CTCs  use  white  boards,  enlarged  sketches 
or  maps  combined  with  participation  feedback  and  open-ended  and  leading  questioning 
techniques.  Many  combat  arms  trainees  below  battalion  level  do  not  have  the  ability  to 
verbally  communicate  clearly.  Subject  matter  experience  has  shown  that  drawing  a 
picture/sketch  or  moving  an  object  on  a  terrain  model  often  helps  soldiers  articulate  their 
point.  Additionally,  the  physical  action  encourages  participation  in  the  discussion.  It 
follows  then,  that  accurate  3-D  terrain  boards  would  also  be  good  tools  to  prompt 
physical  participation  and  focused  discussion.  These  additional  discussion  aids,  with  the 
exception  of  the  large  TV  screen,  need  the  attention  of  R  &  D  in  order  to  make  them 
more  responsive  to  the  trainee  during  discussion. 

There  is  little  doubt  that  AAR  systems  are  being  developed  to  produce  the 
relevant  performance  data  to  support  focused  discussion  points.  But  the  systems  are  not 
flexible  enough  to  aid  the  participants  in  self  critique.  In  other  words,  AAR  systems  need 
the  capability  to  allow  the  participants  to  identify  the  AAR  points  and  then  produce  the 
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relevant  performance  data  that  it  has  recorded.  This  would  empower  the  unit  leaders  by 
giving  them  control  of  the  AAR.  In  turn,  this  empowerment  would  reinforce  the  leader’s 
responsibility  to  supervise  corrective  training  to  standard;  thus,  improving  performance. 

For  most  actions  at  the  platoon  and  company  levels,  the  combination  of  word 
slides  with  AAR  focus  points,  a  video  with  audio  replay  of  each  participant’s  action,  an 
accurate  3-D  terrain  model,  and  a  white/chalk  board  are  the  tools  necessary  to  create  the 
best  conditions  for  the  trainees  to  assimilate  the  feedback.  Yet  none  of  the  AAR  systems 
consider  the  combination  of  products  and/or  more  than  one  presentation  medium.  The 
trainer  using  STAARS  is  allowed  a  choice  of  formats  for  4  of  the  101  AAR  products. 
(BCTD,  1995)  To  design  a  complete  AAR  system,  multiple  presentation  methods  must 
be  integrated  with  the  standardized  product  formats.  An  AAR  system  should  be  designed 
to  integrate  multimedia  presentation  techniques  (TV  screen  with  sound  system,  3-D 
terrain  model,  and  chalk  board)  and  leverage  new  presentation  technologies  (holographic 
replay,  voice-to-text  recording,  etc.). 

Notwithstanding  these  potentials  for  technological  improvement,  the  process  that 
these  improvements  would  support  must  be  sound.  A  lesson  of  project  and 
organizational  engineering  over  the  past  20  years  is  that  technological  advances  or 
improvements  do  not  solve  problems  rooted  in  a  flawed  process.  Multiple  examples  of 
this  can  be  found  in  organizations  that  adopted  the  computer  or  some  form  of  automation 
as  a  solution  to  problems  in  organizational  structure,  strategic  planning,  and  leadership. 
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(Kerzner,  1995,  56, 161, 431,  and  647)  The  basic  process  must  be  effective  before  it  can 
be  made  more  efficient. 

The  majority  of  CTC  AARs  surveyed  were  ineffective  because  AAR  leaders  did 
not  focus  discussion  on  solving  a  performance  problem  completely  nor  did  they  prompt, 
monitor,  and  encourage  trainee  participation  to  do  so.  The  AAR  at  the  CTCs  is  primarily 
a  lecture  presentation  of  OC  observations  and  recommendations  with  little  trainee  input. 
These  are  procedural  deficiencies  that  standardized  information  formats  will  not  solve. 

By  imitation  or  support  of  CTC  AAR  systems,  small  unit  simulation  training 
systems  seek  to  improve  an  ineffective  process.  None  of  the  systems  —  STAARS,  UPAS, 
CCTT,  ATAFS,  and  STRIPES  -  integrate  a  terrain  model,  trainee  drawn  sketch,  or  other 
discussion  media  into  the  AAR.  (Appendix  C,  Current  AAR  Systems  in  Training 
Simulations)  ATAFS  has  the  potential  to  benefit  the  AAR  leader  the  most  with  the 
automated  ouQiut  of  pre-designated  questions.  But  this  increases  the  AAR  leader’s  input 
and  articulation,  not  the  trainees.’  As  currently  designed,  small  unit  simulation  training 
AAR  support  systems  are  inadequate  and  incomplete  for  the  purpose  of  conducting  an 
effective  AAR.  AAR  system  designs  must  consider  the  trainees  and  what  the  AAR 
leader  needs  to  get  them  to  participate  in  a  discussion  of  training  performance.  The 
problem  to  solve  concerns  what  the  trainee  needs  in  order  to  learn  from  his  performance. 
An  AAR  is  not  effective  if  the  trainees  do  not  learn.  The  approach  outlined  in  this 
chapter  offers  a  solution  to  increase  the  trainees’  learning  efficiency. 
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Simulation  training  AAR  systems  are  actually  a  single  component  of  the  AAR 
process  that  support  the  efficient  collection  and  presentation  of  performance  data. 
However,  the  functionality  of  these  AAR  systems  stops  with  the  display  of  performance 
data.  These  systems  are  not  capable  of; 

•  Prompting  trainee  participation  in  a  problem  solving  discussion; 

•  Supporting  trainee  elaboration; 

•  Supporting  retraining  to  reinforce  learning  points. 

Notwithstanding,  what  these  simulation  support  systems  are  capable  of  is  absolutely 
critical  to  the  AAR  process.  These  systems  provide  what  Marshall  sought  on  the 
battlefields  of  Europe,  the  South  Pacific,  and  Vietnam  -  ground  truth.  If  performance  is 
not  improved  with  respect  to  what  actually  happened,  then  mistakes  are  likely  to  be 
repeated  in  the  absence  of  blind  luck. 

The  proposed  solution  to  the  above  problem  entails:  1)  implementing  a  meta- 
cognitive  instructional  theory  to  guide  a  collaborative  discussion  focused  on  solving 
performance  problems;  2)  using  interactive  discussion  mediums  that  support  trainee 
elaboration  and  discussion  (of  the  problem  and  solutions);  3)  using  simulation  technology 
to  reinforce  the  learning  point  (solution  to  the  problem);  and  4)  using  simulation 
technology  to  ensure  the  timely  conduct  of  the  AAR  and  learning  reinforcement 
exercises. 
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The  novelty  of  this  approach  lies  in  the  detailed  analysis  of  effectiveness  in  the 
AAR  and  the  extension  of  the  AAR  process  to  encompass  re-training.^^  Simulation 
training  systems  and  AAR  subsystems  have  allowed  three  of  the  elements  of 
effectiveness  (discussion  focus,  learning  reinforcement,  and  timeliness)  to  be  realized.  A 
proposed  design  concept  based  upon  this  approach  is  presented  in  Appendix  E,  Proposed 
Design. 

The  proposed  AAR  approach  implements  Allan  Collins’  Inquiry  theory  to 
increase  the  effectiveness  and  efficiency  of  the  trainees’  learning.^''  (Collins  &  Stevens, 
1983)  This  instructional  strategy  increases  learning  effectiveness  by  requiring 
participants  to  fully  explain  what  happened,  why  it  happened,  and  how  to  perform  the 
task(s)  better.  None  of  these  elements  are  allowed  to  be  ignored.  Learning  efficiency  is 
increased  by  guiding  the  participants  to  discuss  only  the  cause  and  effect  relationships 
that  determine  the  performance  outcome.  This  guidance  is  logically  structured  and 
closely  resembles  the  scientific  method  -  systematically  formulating  and  testing  multiple 
hypotheses.  Extraneous  and  distracting  subjects  are  avoided  through  the  theory’s 
structured  approach  to  problem  solving. 


The  term  “retraining”  denotes  the  training  that  occurs  after  a  training  exercise  to  correct  deficiencies 
uncovered  during  the  exercise  and  AAR. 

Learning  effectiveness  and  efficiency  are  covered  in  Chapter  II,  45, 54  and  55. 
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Inquiry  Theory 


Allan  Collins,  a  leading  scientist  in  the  fields  of  cognitiye  science  and  human 
semantic  processing,  deyeloped  an  instructional  theory  that  is  labeled  inquiry  teaching. 
(Collins  &  Steyens,  1983, 247  to  248)  Collins  inductiyely  deyeloped  this  theory  by 
obserying  a  number  of  expert  teachers,  cataloging  the  instructional  strategies  they  used  to 
teach  concepts  and  ideas  in  differing  situations,  and  deyeloping  heuristics  for  the 
application  of  each  strategy.  (Collins,  1987, 181)  With  the  aim  of  teaching  students  how 
to  solye  problems  more  efficiently  and  effectiyely.  Inquiry  theory  is  ideal  for  knowledge 
acquisition.  The  theory  has  two  oyerarching  goals:  to  teach  students  1)  how  a  specific 
rule  or  theory  is  applied,  and  2)  how  to  deriye  a  rule  or  theory.  (Collins  &  Steyens,  1983, 
249) 

Inquiry  theory  fits  knowledge  domains  with  causal  relationship  structures.  This  is 
demonstrated  by  the  wide  yariety  of  subject  domains  Collins’  subjects  were  teaching 
when  the  data  for  this  theory  was  gathered.  These  disciplines  were;  arithmetic,  art 
history,  law,  medicine,  geography,  moral  education,  botany,  and  computer  science. 
(Collins  &  Steyens,  1983, 251  to  257)  In  art  history,  the  problem  spaces  that  are 
examined  are  pictures  and  sculptures.  The  causes  and  effect  are  separated.  Painting 
techniques  and  different  of  parts  of  the  painting  interrelate  to  produce  the  effect,  in  this 
case,  on  the  yiewer.  Another  example  can  be  shown  with  medicine.  The  problem  spaces, 
or  “cases”  as  Collins  calls  them,  are  medical  cases.  The  symptoms,  history,  and  course  of 


symptoms  are  the  factors  that  combine  to  cause  a  disease  -  the  effect.  (Collins  &  Stevens, 
1983, 254) 

Inquiry  theory  fits  well  in  the  domain  of  Army  doctrine  and  AARs.  Essentially, 
the  AAR  is  founded  in  doctrine  -  which  is  a  set  of  principles  and  application  rules  (for 
those  principles).  During  training,  the  trainee  applies  these  principles  to  different 
situations.  The  key  to  training  performance  diagnosis  is  to  discover  the  cause,  or 
combination  of  causes,  of  a  specific  performance  output.  Hence,  the  cases  (problem 
spaces)  that  must  be  diagnosed  are  missions,  situational  training  exercises,  or  a  phase  of 
mission  execution.  The  effect  is  the  performance  output  or  result  identified  as 
imdesirable.  These  are  judged  xmdesirable  with  respect  to  the  performance  standards 
outlined  in  the  mission  training  plan  (MTP)  references  or  the  unit’s  organizational 
procedures,  objectives,  and  goals.  The  causes  of  the  effect  are;  1)  the  lower  order 
supporting  MTP  subtasks  and  performance  requirements,  and  2)  the  unit’s  analysis  and 
actions  with  respect  to  the  mission,  enemy,  terrain,  troops,  and  time  conditions.  The 
Inquiry  approach  efficiently  and  systematically  leads  the  analyst  through  the  great 
number  of  potential  causes  and  combinations  of  causes  to  identify  those  that  specifically 
affected  the  outcome. 


Inquiry  Theory  Components 

Inquiry  theory  has  three  components.  These  components  are:  the  goals  and 
subgoals  of  the  teacher,  the  dialogue  strategies  teachers  use  to  achieve  the  goals,  and  the 
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control  structure  the  teacher  uses  to  allocate  time  pursuing  different  goals  and  subgoals. 
(Collins  &  Stevens,  1983, 257, 260,  and  274) 

It  is  helpful  to  understand  the  causal  structure  notation  that  Collins  uses  before 
tackling  the  theory.  Collins  represents  causal  dependencies  in  terms  of  variables/factors, 
rules,  and  theories.  (Collins  &  Stevens,  1983, 251  to  253)  The  factors  are  categorized  as 
dependent  and  independent  variables.  Whether  a  variable  is  classified  as  dependent  or 
independent  depends  upon  the  goal/subgoal  of  the  teacher  without  respect  to  the  direction 
of  causality  between  the  variables.  The  dependent  variable  is  “what  one  tries  to  make 
predictions  about  in  the  real  world.”  (Collins  &  Stevens,  1983, 255) 

Given  a  set  of  factors  and  a  dependent  variable,  a  rule  describes  the  conditional 
relationship  of  one  or  more  factor  values  to  dependent  variable  values.  (Collins  & 
Stevens,  1983, 252)  A  rule’s  utility  or  value  is  judged  on  how  well  it  accounts  for  the 
independent  variables  that  affect  the  dependent  variable  value  range.  In  turn,  a  theory 
specifies  the  causal  structure  of  interrelated  rules.  (Collins  &  Stevens,  1983, 252)  The 
theory’s  utility  is  judged  on  the  number  of  situations  in  which  it  can  be  successfully 
applied  to  a  specific  problem  domain. 

Graphically,  the  factor  relationships  that  correspond  to  a  rule  or  theory  can  be 
shown  as  a  task  tree  or  and/or  graph.  Figure  4. 1 ,  Control  Organic  Fires  (next  page), 
depicts  the  factor  relationships  for  that  leader  task.  This  task  supports  any  collective  task 
{Assault  or  Overwatch/Support  by  Fire  for  example)  that  requires  an  enemy  to  be 
engaged  with  direct  fire  weapons.  (DA,  ARTEP  7-8-MTP,  1994) 
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Figure  4.1,  Control  Organic  Fires 

Note:  AND/OR  Graph  for  Control  Organic  Fires 

AND=  "  "  OR=" _ 

Un-circled  subtasks  are  not  expanded. 
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Goals  and  Subgoals 


As  stated  before,  the  two  top-level  goals  of  inquiry  teaching  are  to  teach  the 
student  1)  specific  rules/theories,  and  2)  how  to  derive  rules/theories.  (Collins  &  Stevens, 
1983, 257)  Collins  associates  two  subgoals  with  the  first  top-level  goal  and  four  subgoals 
with  the  second.  These  subgoals  are  the  intermediate  objectives  that  must  be  achieved  in 
order  to  satisfy  the  top-level  goals.  They  are  not  comprehensive,  but  represent  the 
subgoals  that  were  most  frequently  employed  by  the  expert  teachers  Collins  researched. 
(Collins  &  Stevens,  1983, 258) 

The  subgoals  that  support  the  teaching  of  a  specific  rule/theory  are:  1)  the 
debugging  of  incorrect  hypotheses,  and  2)  the  teaching  of  how  to  make  novel  predictions 
based  upon  a  specific  rule/theory.  The  student  must  learn  to  analyze  a  hypothesis  to 
determine  if  it  is  correct  or  incorrect  and,  if  incorrect,  identify  the  false  assumption, 
premise,  or  factor.  This  debugging  is  critical  in  preventing  students  fi:om  forming 
misconceptions  about  the  application  of  the  rule/theory.  (Collins  &  Stevens,  1983, 258) 

The  second  subgoal,  learning  how  to  make  novel  predictions  with  a  rule/theory, 
follows  from  the  first.  Simply,  this  subgoal  requires  the  student  to  know  how  to  properly 
apply  the  rule/theory  to  a  unique  set  of  conditions  and  accurately  predict  an  outcome. 
(Collins  &  Stevens,  1983, 258)  This  requires  the  student  to  fully  understand  the  factors, 
interrelationships,  and  overall  structure  of  the  rule/theory.  They  must  apply  the 
rule/theory  purportedly  learned. 
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The  four  subgoals  associated  with  learning  how  to  derive  a  rule/theory  are 
learning:  1)  what  questions  to  ask  in  order  to  derive  the  (factors  and  assumptions  of  the) 
rule/theory,  2)  what  form  a  rule/theory  should  take,  3)  how  to  evaluate  a  rule/theory  just 
constructed,  and  4)  how  to  verbalize  and  defend  the  rule/theory  just  constructed.  (Collins 
Stevens,  1983, 259)  Many  of  these  subgoals  depend  upon  the  previous  set  of  subgoals 
-  the  abilities  to  debug  incorrect  rules/theories  and  apply  rules/theories  to  make  a 
prediction. 

Learning  what  questions  to  ask  in  order  to  derive  a  new  rule/theory  and  what 
form  (cause  and  effect  structure)  a  rule/theory  should  take  go  hand  in  hand.  To  meet 
these  two  subgoals,  there  are  three  basic  questions  that  the  student  needs  to  answer, 
irrespective  of  subject  domain.  In  order,  these  are: 

1.  What  is  the  dependent  variable  that  the  rule/theory  must  describe? 

2.  What  are  the  factors  that  affect  the  dependent  variable? 

3.  How  do  these  factors  relate  to  each  other  to  affect  the  dependent  variable? 

The  second  question  should  be  repeated  for  the  (just  identified)  factors  that  affect  the 
dependent  variable  so  that,  through  multiple  iterations,  most  of  the  important  factors  are 
identified.  Answering  these  questions  in  this  sequence  will  aid  the  student  in  describing 
the  causal  relationship  structure  of  the  rule/theory. 

Learning  how  to  evaluate  a  freshly  constructed  rule/theory  is  dependent  upon  the 
domain  specific,  evaluation  criteria.  (Collins  &  Stevens,  1983, 259)  For  performance 
analysis  in  the  AAR  environment,  trainees  may  use  logic/common  sense,  doctrinal 
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principles  and  training  standards,  past  experiences  of  individuals,  and/or  simulations  to 
apply  the  rule/theory  to  a  new  set  of  battlefield  conditions  (METT-T).  This  step  is 
critical  in  defining  which  conditions  the  rule/theory  is  operable  in.  A  key  characteristic 
that  soldiers  must  realize  is  that  the  derived  rule/theory  is  not  a  stand-alone  principle,  but 
a  condition  specific  phenomenon.  The  mission,  enemy,  terrain,  troops,  and  time 
conditions  dictate  the  utility  of  the  rule/theory.  The  variations  of  each  of  these  conditions 
affect  the  dependent  and  independent  variables;  thus  making  the  combat  performance 
domain  extremely  complex. 

The  last  subgoal,  teaching  the  student  how  to  verbalize  and  defend  a  rule/theory, 
is  critical  to  learning  and  achieving  a  common  mental  model  among  a  group.  (Chapter  II) 
The  verbalization  of  the  reasoning  behind  a  rule/theory  causes  the  student  to  articulate 
and  elaborate  the  important  factor  relationships  and  conditions  that  affect  the  dependent 
variable.  The  elaboration  is  a  form  of  practice  that  constructs  and  reinforces  production 
rules.  (Anderson,  1993)  In  the  production  system  theory  of  cognition  (Anderson,  1993), 
knowledge  and  skill  are  represented  by  production  rules  (condition-action  pairs)  and 
chunks  (groups  of  production  rules).  Elaboration  plays  a  major  role  in  strengthening  the 
connection  between  and  within  productions  and  chunks.  (Williams,  1996a)  This 
connection  strength  determines  what  one  knows  and  is  able  to  recall  for  use.  (Anderson, 
1993)  Without  the  connection,  the  student  cannot  know  and/or  apply  the  rule/theory. 
Thus,  what  you  learn  is  based  upon  what  you  can  relate  to  previous  experience;  given  that 
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previous  experience  consists  of  knowledge  that  exists  as  productions  and  chunks  in 
procedural  and  declarative  memory.  (Appendix  A,  Theoretical  Formdations) 

The  oral  defense  of  the  rule/theory  in  the  face  of  criticism  motivates  the  student  to 
articulate  and  elaborate  his  reasoning  as  competently  as  possible,  a  natural  result  of 
perceived  peer  pressure.  To  prompt  this  subgoal,  Collins  recommends  requiring  each 
student  to  either  articulate  and  elaborate  the  reasoning  for  his  own  rule/theory  or  support 
or  criticize  the  reasoning  of  another.  (Collins  &  Stevens,  1983, 259)  This  discussion 
results  in  the  rule/theory  being  understood  by  the  group.  They  then  share  a  common 
understanding  of  the  subject  matter  and  can  collectively  work  to  improve  performance  in 
this  context. 

Strategies 

Given  these  goals,  the  teacher  constantly  assesses  and  diagnoses  each  student’s 
domain  knowledge  with  respect  to  his  line  of  reasoning.  The  teacher  selects  one  of  10 
strategies  with  which  to  engage  the  student  in  order  to  achieve  one  of  the  two  goals  stated 
above.  The  strategy  selection  is  based  upon  what  the  student  is  trying  to  learn  (goal  or 
subgoal),  what  he  knows  and  does  not  know,  and  his  misconceptions  about  how 
rules/theories  are  formed  and  applied.  (Collins  &  Stevens,  1983, 251  to  257,  and  Collins, 
1987, 183  and  195  to  198)  However,  all  of  these  techniques  force  the  student  to  elaborate 
and,  thus,  learn. 
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Collins  outlines  10  strategies  that  are  commonly  used  by  expert  teachers  to  teach 
students. 

1.  Selecting  positive  and  negative  exemplars  is  used  to  demonstrate  the 
relationship  between  relevant  factors  in  a  concept,  rule,  or  theory.  The  chosen  examples 
are  often  positive  or  negative  paradigm  cases  for  that  concept,  rule,  or  theory  in  that  all 
the  major  factors  consistently  lead  to  a  conclusion  that  is  in  line  with  the  concept,  rule,  or 
theory,  (Collins  &  Stevens,  1983, 260,  and  Collins,  1987, 183) 

Collins  illustrates  this  strategy  for  teaching  what  geographical  factors  affect 
rainfall.  As  positive  exemplars  demonstrating  factors  that  lead  to  heavy  rainfall,  the  cases 
of  the  Amazon,  Oregon,  and  Ireland  are  examined.  The  negative  exemplars  that 
demonstrate  factors  that  cause  little  rainfall  are  southern  California,  northern  Afiica,  and 
northern  Chile.  Once  the  students  imderstand  the  faetor  relationships  of  rainfall,  the 
teacher  covers  more  complex  factor  interactions  that  affect  rainfall  in  areas  such  as  the 
eastern  United  States  or  China.  (Collins  &  Stevens,  1983, 260  to  261) 

This  strategy  can  be  used  to  illustrate  the  terrain  factors  that  impact  on  effectively 
suppressing  an  enemy  position  from  a  support  by  fire  position.^^  Examples  of  support  by 
fire  positions  with  cover  and  concealment  can  be  compared  to  positions  without  cover 
and  concealment.  In  this  example,  the  trainees  should  be  able  to  discern  what  constitutes 
cover  and  concealment  for  the  task  given  a  specific  terrain  type. 


”  An  enemy  position,  aka  an  “objective”,  is  an  enemy  unit  (group)  occupying  a  specific  area  of  ground 
(terrain).  Tbus  an  enemy  position  actually  consists  of  a  number  of  positions  with  individual  or  pairs  of 
individuals. 
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2.  Varying  cases  systematically  is  a  strategy  that  is  used  to  highlight  various 
interactions  between  a  dependent  variable  and  different  factors  of  a  concept,  rule,  or 
theory.  The  systematic  sequence  of  example  variation  can  emphasize  how  a  particular 
factor  relates  to  changing  conditions  and  factors  or  a  combination  of  factors.  (Collins  & 
Stevens,  1983, 262,  and  Collins,  1987,  183  to  184) 

Collins  gives  an  example  of  this  strategy  for  teaching  a  student  about  the 
properties  of  light  rays  shining  through  a  lens.  Given  a  light  source  and  a  lens,  the 
distance  between  the  lens  and  the  light  source  can  be  systematically  varied  to  demonstrate 
how  light  rays  cross  over,  come  to  a  focal  point,  and  diverge,  (as  they  travel  from  the 
source,  through  a  lens,  to  a  reflecting  object).  (Collins,  1987, 183  to  184) 

Using  the  same  support  by  fire  position  example,  different  cover  and  concealment 
combinations  may  be  used  to  highlight  the  affect  on  communication  within  the  friendly 
element  The  ease  of  communication  amongst  the  element  varies  inversely  with  the 
amoimt  of  cover  and  concealment  the  unit  takes/assiunes.  Consequently,  the  realization 
of  this  relationship  impacts  on  how  the  element’s  fires  are  controlled  and  distributed.^* 
Here  the  AAR  leader  must  show  an  example  of  the  unit  in  a  support  position  with  good 
observation  and  fields  of  fire.  The  number  and  location  of  enemy  hit/missed  and  number 
and  type  of  rounds  fired  at  each  enemy  position  are  the  required  BDA.  In  the  absence  of 
fire  control  measures,  big  and  close  targets  located  in  the  middle  of  the  engagement  area 


Fires  (noun)  is  the  act  and  effect  of  firing  a  weapon(s). 
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are  usually  hit  first.  This  forces  the  trainees  to  reform  the  rule  for  effective  suppression  of 
an  enemy  to  include  fire  control. 

3.  Selecting  counterexamples  strategy  is  used  when  a  student  forms  an 
incomplete  or  incorrect  hypothesis  about  the  concept,  rule,  or  theory.  In  this  case,  the 
teacher  selects  an  example  that  satisfies  the  student’s  misconception  but  violates  the 
actual  concept,  rule,  or  theory.  (Collins  &  Stevens,  1983, 263  to  264,  and  Collins,  1987, 

1 84)  Reigelufii,  the  editor  of  Collins  (1987),  points  out  that  this  strategy  is  likely  to 
contribute  to  deep  cognitive  processing  (understanding)  of  the  concept,  rule,  or  theory. 
(Reigeluth,  1987, 184)  However,  he  does  not  explain  why  this  is  so. 

Collins  demonstrates  this  strategy  with  a  student  who  may  incorrectly  believe  that 
a  magnifying  glass  always  makes  objects  such  as  letters  printed  on  a  sheet  of  paper  larger. 
To  counter  this  hypothesis,  the  teacher  may  move  the  magnifying  glass  to  a  point  that  is 
halfway  between  the  print  and  the  student’s  eye  and  demonstrate  otherwise  to  him. 
(Collins,  1987,  184) 

A  supporting  element  may  ignore  the  requirement  for  fire  control  measures  and 
equate  good  observation  and  fields  of  fire  with  effective  suppression  of  an  enemy 
position.^’  In  this  case,  the  counterexample  must  show  enemy  positions  not  being 
effectively  suppressed  when  observation  and  fields  or  fire  are  present. 

4.  Generating  hypothetical  cases  is  used  to  force  students  to  reason  about  an 
aspect  of  a  concept,  rule,  or  theory  and  generalize  this  aspect  to  differing  situations.  This 
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strategy  is  similar  to  the  strategy  of  producing  coimterexamples  since  its  aim  is  to  test  the 
student’s  knowledge  of  and  ability  to  apply  the  concept,  rule,  or  theory.  (Collins  & 
Stevens,  1983, 264  and  265,  and  Collins,  1987, 184) 

Collins  demonstrates  this  strategy  by  challenging  a  geography  student’s  ability  to 
determine  why  a  specific  area,  such  as  Louisiana,  would  support  rice  cultivation.  He 
does  this  by  supposing  how  rice  might  grow  in  Louisiana  without  the  rainfall. 

For  this  strategy  the  AAR  leader  can  challenge  the  trainees  to  explain  how  they 
would  alter  fire  control  methods  to  achieve  effective  suppression  of  an  enemy  in  differing 
situations.  Some  examples  of  differing  situations  are: 

•  Given  an  enemy  unit  occupying  hasty  positions;  in  sparsely  wooded,  flat 

terrain;  in  daylight  with  unaided  observation  <100  meters  and  restricted 
by  ground  vegetation. 

•  Given  an  enemy  unit  occupying  hasty  positions;  in  rocky,  moimtainous  terrain; 

in  daylight  with  unaided  observation  <  1000  meters  and  not  restricted  by 
ground  vegetation. 

•  Given  an  enemy  unit  occupying  prepared  positions;  in  sparsely  wooded,  rolling 

terrain;  in  limited  visibility  with  xmaided  observation  <  25  meters, 
restricted  by  ground  vegetation,  and  50%  lunar  illumination. 


”  The  performance  standard  for  “effective  suppression”  is  defined  as  the  platoon  engaging  at  least  50%  of 
the  enemy  position  and  destroying  or  forcing  the  withdrawl  of  100%  of  enemy  in  an  assigned  engagement 
area.  (DA,  ARTEP  7-8-MTP,  1988,  5-9) 
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This  forces  the  trainees  to  articulate  how  fire  control  measures  are  affected  by  specific 
terrain  and  weather  factors.  It  is  not  enough  for  a  trainee  to  simply  list  the  capabilities  of 
his  equipment  and  weapons,  he  must  understand:  1)  how  to  apply  those  capabilities  and 
2)  which  capability  is  maximally  effective  in  a  situation, 

5.  Forming  hypotheses  is  the  most  prevalent  strategy  used  by  teachers.  They  get 
students  to  formulate  general  rules  about  different  factors  in  relation  to  various  values  of 
a  dependent  variable.  (Collins  &  Stevens,  1983, 265,  and  Collins,  1987,  184)  This 
strategy  is  aimed  at  Collins’  second  goal  -  getting  students  to  derive  their  own  rule  or 
theory  given  some  evidence.  (Editor  of  Collins,  1987, 184) 

Collins  illustrates  diis  strategy  with  a  teacher  asking  a  student  to  formulate  a  set  of 
rules  that  describe  what  focal  length  is  dependent  upon.  The  student  hypothesizes  that 
the  curvature  of  a  lens  inversely  affects  focal  length;  i.e.,  the  greater  the  curvature,  the 
less  the  focal  length.  (Collins,  1987,  184) 

The  AAR  leader  can  ask  trainees  to  hypothesize  rules  to  determine  when  a 
specific  type  of  marking  signal  should  be  used  to  control  the  organic  fires  of  the  unit. 
Marking  signals  can  generally  be  categorized  as  visual,  aural,  and  tactile  signals  to  mark 
enemy,  friendly,  or  target  reference  point  positions,^* 

6.  Testing  hypotheses  is  a  strategy  in  which  the  teacher  tries  to  get  the  student  to 
systematically  test  a  hypothesis.  Essentially,  the  student  must  learn  to  identify  the 
different  variables  and  then  test  a  dependent  variable  while  holding  all  the  other  variables 

A  target  reference  point  (TRP)  is  any  point  that  is  known  to  the  unit. 
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constant.  (Collins  &  Stevens,  1983, 267  to  268,  and  Collins,  1987,  184)  This  is  an 
extension  of  the  teacher  varying  cases  systematically,  the  second  strategy,  which  serves 
as  an  example  for  the  student. 

Collins  demonstrates  this  strategy  with  a  continuation  of  the  example  used  in 
forming  a  hypothesis  above.  After  the  student  formulates  his  hypothesis  about  what 
factor(s)  affect  focal  length,  the  teacher  asks  him  how  the  hypothesis  can  be  tested.  The 
point  is  to  make  the  student  derive  the  requirements  that  will  lead  to  a  sound  test  of  the 
hypothesized  factor,  in  this  case,  the  curvature  of  the  lens.  The  control  of  different 
variables  and  testing  of  special  cases  are  examples  of  the  requirements  the  student  must 
account  for.  (Collins  &  Stevens,  1983, 267,  and  Collins,  1987,  184) 

After  the  trainee  has  formulated  a  general  rule  for  employing  marking  signals,  the 
AAR  leader  asks  him  to  give  an  example  of  how  he  could  have  employed  the  rule  in  the 
training  exercise  the  AAR  is  examining.  The  key  is  to  force  the  trainee  to  articulate  the 
factors  (that  were  present)  that  should  have  been  considered. 

7.  Considering  alternative  predictions  is  a  strategy  in  which  the  teacher 
encourages  the  student  to  consider  different  alternative  values  of  a  dependent  variable. 
The  teacher  is  trying  to  get  the  student  to  consider  how  the  values  of  the  known  factors 
relate  to  other  possible  dependent  variable  values.  (Collins  &  Stevens,  1983, 268)  This 
fosters  differential  diagnosis  or  comparative  hypothesis  testing  as  a  learning  strategy. 
(Collins  &  Stevens,  1983, 269,  and  Collins,  1987, 184) 
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“For  example,  if  a  student  thinks  that  an  image  turns  upside  down  as  one  moves  a 
lens  toward  one’s  eye,  the  tutor  might  ask  the  student  to  consider  whether  instead  it  might 
turn  the  image  right  side  up.”  (Collins,  1987, 184) 

To  encourage  the  trainees  to  consider  alternative  predictions,  the  AAR  leader  may 
ask  if  it  might  be  better  to  mark  a  TRP  rather  than  a  friendly  position  in  order  to  orient 
friendly  fires  on  to  an  enemy.  Another  possibility  is  to  ask  the  trainees  to  describe  what 
their  actions  should  look  like  from  the  enemy  position  and  consider  how  this  perspective 
affects  the  hypothesized  rule. 

8.  Entrapping  students  is  a  strategy  in  which  the  teacher  offers  incorrect 
hypotheses  in  order  to  get  students  to  reveal  their  underlying  misconceptions.  Here  the 
teacher  is  taking  the  student’s  reasoning  and  turning  it  into  a  general  rule  and  then 
offering  a  counterexample  on  which  to  test  the  rule.  The  carefully  chosen 
counterexample  can  reveal  faulty  reasoning  based  upon  unnecessary  or  insufficient 
factors,  and  incorrect  factor  relationship  interpretation.  This  leads  the  student  to  learn 
more  complex  relationships,  after  positive  and  negative  exemplars  are  established,  and 
understand  the  subject  matter  more  deeply.  (Collins  &  Stevens,  1983,  270  to  271,  and 
Collins,  1987, 185) 

Collins  demonstrates  this  strategy  with  a  student  who  is  incorrectly  using  the 
equatorial  latitude  as  the  only  factor  to  predict  surface  temperature.  Collins  chooses  a 
counterexample  of  equatorial  Peru.  At  this  location,  the  temperature  is  much  cooler  than 
was  induced  because  of  an  ocean  current.  (Collins  &  Stevens,  1983, 270) 
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For  trainees  who  incorrectly  assume  that  they  should  never  mark  the  fnendly  unit, 
the  AAR  leader  could  entrap  them  by  giving  them  a  situation  in  which  different  people  in 
the  element  can  see  different  parts  of  the  enemy,  but  none  can  see  the  whole  objective. 
Adding  an  assault  element  (that  the  overwatch  element  is  supporting)  and  TRP’s  that 
cannot  be  designated  or  seen  by  all  of  the  overwatching  element,  the  tactical  solution  is  to 
mark  the  friendly  assault  element  and  adjust  the  suppressive  fires  from  them. 

9.  Tracing  consequences  to  a  contradiction  is  a  strategy  the  teacher  uses  to  get 
the  student  to  correct  his  misconceptions.  Given  a  student’s  incorrect  belief,  the  teacher 
traces  the  consequences  of  the  misconception  to  a  conclusion  that  is  plainly  incorrect. 
This  is  a  form  of  immediate  feedback  given  on  the  practice  of  formulating  hypotheses 
(Reigeluth,  1983, 271)  and  forces  the  student  to  consider  the  implications  of  his 
predictions  on  all  of  the  factors.  Hence,  they  build  consistent  theories  and  learn  to  debug 
these  theories.  (Collins  &  Stevens,  1983, 271  to  273,  and  Collins,  1987,  185) 

Collins  demonstrates  this  strategy  with  a  student  that  has  incorrectly  ascertained 
that  evaporation  from  the  Amazon  River  is  responsible  for  the  heavy  rainfall  in  the 
region.  In  fact,  evaporation  from  the  ocean  is  the  largest  factor  of  heavy  rainfall.  Collins 
points  this  out  with  the  following  questions:  “1)  Does  most  of  the  water  in  the  river 
evaporate  or  flow  into  the  ocean?  2)  If  most  of  the  water  flows  into  the  ocean,  won’t  the 
process  soon  dry  up?”  The  student  is  then  forced  to  realize  that  evaporation  from  the 
ocean  is  responsible  for  the  heavy  rainfall  in  the  Amazon  region.  (Collins  &  Stevens, 
1983, 273) 
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Again,  given  a  unit  that  believes  that  visually  marking  the  enemy  is  the  only  fire 
control  marking  method  they  need,  the  AAR  leader  may  ask  them  how  they  know  where 
to  shoot  if  they  cannot  see  the  marker.  He  can  then  ask  them  if  they  will  move  (and 
expose  themselves)  or  not  shoot  at  all?  The  trainees  should  then  realize  that  they  must 
choose  one  of  the  following  methods:  mark  a  TRP  (to  reference  and  adjust  from),  mark 
the  friendly  assault  element  as  a  no  fire  zone,  or  employ  a  combination  of  marking 
signals. 

10.  Questioning  authority  is  a  strategy  in  which  the  teachers  try  to  make  the 
students  self-reliant  in  constructing  their  own  theories  and  conclusions  and  not  relying  on 
the  teacher  or  a  generalized  book  solution.  Here,  the  teacher  feigns  ignorance  or  simply 
refuses  to  give  the  student  an  answer.  This  is  an  important  strategy  in  motivating 
hypothesis  formulation  and  testing  and  influencing  the  student  to  question  assumptions 
and  facts  that  appear  to  be  givens  (Collins  &  Stevens,  1983, 273  to  274,  and  Collins, 
1987, 185) 

Collins  demonstrates  this  strategy  with  a  student  who  asks  whether  a  lens  that  is 
more  curved  has  a  shorter  focal  length  than  a  lens  that  is  less  curved.  In  this  case,  the 
teacher  can  ask  how  the  student  can  construct  an  experiment  to  find  out  for  himself  and 
then  tell  him  to  do  it.  (Collins,  1987,  185) 

This  should  be  a  well  used  strategy  for  AAR  leaders  to  prompt  participation.  He 
can  accomplish  this  by  feigning  ignorance  or  refusing  to  solve  their  problem.  Whichever 
the  AAR  leader  chooses,  the  trainees  must  be  required  to  solve  the  problem. 
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All  ten  of  these  strategies  are  based  upon  the  teacher:  1)  knowing  the  body  of 
knowledge  (subject  matter)  that  must  be  taught  -  having  a  lesson  plan,  and  2)  having  a 
good  deal  of  classroom  time  to  pursue  these  strategies.  The  AAR  leader-trainee 
relationship  is  not  a  classic,  teacher— student  relationship.  There  are  a  number  of 
differences  between  the  classic  teacher-student  instruction  and  the  AAR  leader-trainee, 
not  least  of  which  are  the  basic  assumptions  of  each. 

A  second  difference  lies  in  the  AAR  leader’s  role  and  authority.  A  tutor’s 
response  to  a  student’s  impasse  intends  to  focus  the  student  back  to  the  supposed  correct 
path  of  learning.  (Chan  and  Baskin,  1990,  8)  But  in  AAR  cases,  the  leader  may  not 
always  know  the  correct  paths  or  have  the  authority  to  impose  a  correct  solution.  Unit 
leaders,  who  have  the  authority  to  implement  the  correct  solution,  are  often  inexperienced 
and/or  immature.  CTC  OCs  with  an  expanded  experience  base  are  more  likely  to  have  a 
correct  (or  improved)  solution,  but  lack  the  authority  to  implement  it.  An  additional 
problem  that  could  complicate  the  OCs  role  as  an  AAR  leader  is  the  unit  is  too 
inexperienced  to  realize  the  full  implications  of  the  feedback.  (Chapter  II,  40, 
SHERIKON,  1996,  Anderson,  1993,  Newell,  1992) 

Dialogue  Control  Structure 

The  control  structure  consists  of  four  basic  parts:  1)  a  set  of  strategies  for 
selecting  cases  with  respect  to  the  top-level  goals;  2)  the  student  model  of  what  he  knows, 
does  not  know,  and  misunderstands;  3)  the  agenda  of  goals  and  subgoals;  and  4)  a  set  of 
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priority  rules  for  adding/deleting  goals  and  subgoals  to/ffom  the  agenda.  (Collins  & 
Stevens,  1983, 274)  Overall,  these  parts  collectively  form  the  heuristics  that  the  teacher 
must  learn  in  order  to  help  the  students  achieve  the  top  level  goals. 

The  case  selection  strategies  are  a  set  of  criteria  to  select  a  single  or  set  of  cases 
(examples)  to  optimize  the  student’s  ability  to  master  the  top-level  goals  and  subgoals. 
(Collins  &  Stevens,  1983, 274)  The  first  is  selecting  cases  that  illustrate  more  important 
factors  before  less  important  factors.  Obviously,  this  places  emphasis  on  the  basic 
factors  that  define  the  rule/theory.  Selecting  positive  and  negative  exemplars  and  varying 
cases  systematically  are  strategies  most  often  used  in  these  cases.  (Collins  Sc  Stevens, 
1983, 260  to  261,  and  274) 

The  second  selection  criteria  is  select  cases  to  move  from  concrete  to  abstract 
factors.  (Collins  &  Stevens,  1983, 274)  Students  learn  more  efficiently  if  the  factors 
under  discussion  can  be  related  to  experience.  (Anderson,  1993)  Abstract  factors  require 
the  student  to  generalize  his  experience  and  take  longer  to  learn.  (Collins  &  Stevens, 
1983, 274) 

The  third  case  selection  criteria  is  select  more  important  or  more  frequent  cases 
before  less  important  or  less  frequent  cases.  (Collins  &  Stevens,  1983, 274)  Collins 
observed  that,  other  criteria  being  equal,  the  expert  inquiry  teacher  will  select  cases  that 
are  more  familiar  to  the  student  or  occur  most  often.  (Collins  &  Stevens,  1983, 274) 

The  AAR  leader  can  use  the  case  selection  strategies  to  guide  the  preparation  of 
the  AAR.  The  AARs  first  responsibility  is  to  identify  the  problem  (spaces)  that  the 
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discussion  will  be  focused  on.  For  homestation  training  where  the  AAR  leader  is  a  unit 
leader,  the  problem  spaces  will  be  defined  by  the  training  objectives  of  the  unit.  The 
unit’s  training  objectives  should  have  associated  and  specific  performance  standards.  The 
standards  that  were  not  attained  define  the  problems  that  require  a  solution.  For  CTC  (or 
EXEVAL)  AAR  leaders  who  are  unfamiliar  with  the  trainees’  strengths  and  weaknesses, 
this  is  difficult.  Those  AAR  leaders  must  consider  the  rotational  guidance  issued  by  their 
higher  headquarters  and  their  subjective  assessment  of  the  unit’s  training  proficiency 
level. 

Once  the  problems  are  identified,  the  AAR  leader  selects  examples  (cases)  in 
accordance  with  the  selection  criteria  outlined  above.  Current  simulation  training  system 
AAR  components  (Appendix  C,  Current  AAR  Systems  in  Training  Simulations)  are 
being  designed  to  support  the  selection  and  presentation  of  these  examples.  The  AAR 
leaders  without  the  benefit  of  simulation  system  support,  must  rely  on  hand  drawn 
sketches,  scaled  terrain  models,  and  enlarged  maps  to  implement  the  strategies  and 
facilitate  trainee  elaboration  and  discussion. 

The  second  component  of  the  control  structure  is  the  creation  of  a  student  model. 
The  student  model  is  constructed  from  what  the  teacher  expects  the  student  to  know  and 
the  student’s  performance  during  the  session  dialogue.  As  the  teacher  questions  the 
students  about  the  values  and  relationships  of  independent  and  dependent  variables,  he 
notes  the  factors  the  student  knows/does  not  know  and  the  misconceptions  of  factors  and 
their  relationships.  (Collins  &  Stevens,  1983, 275)  The  critical  function  of  the  student 
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model  is  to  identify  “bugs”  in  the  student’s  reasoning.  “Bugs”  are  errors  or  omissions  in 
the  students’  reasoning  that  prevent  him  from  learning  a  rule/theory  or  how  to  derive  a 
rule/theory. 

In  an  independent  analysis,  Littman  et  al.  (1990)  statistically  verified  Collins’ 
strategies  to  identify  student  bugs.  Their  study  of  1 1  experienced,  collegiate  tutors 
identified  the  four  most  common  strategies  used.  In  order,  they  were:  1)  generating 
hypothetical  cases,  2)  forming  hypotheses,  3)  testing  hypotheses,  and  4)  questioning 
authority.  The  questioning  authority  strategy  was  employed  for  a  secondary  purpose  also 
-to  allow/require  students  to  practice  their  problem  solving  skills.  (Littman  et  al.,  1990) 

The  AAR  leader’s  construction  of  the  student  model  is  his  assessment  of  the 
unit’s  knowledge  and  proficiency  level.  This  assessment  is  done  during  the  AAR 
discussion.  However,  the  AAR  leader  constructs  this  assessment  with  respect  to  doctrine. 
In  other  words,  he  must  be  well  versed  in  the  doctrine  associated  with  the  problem  in 
order  to  realize  what  the  unit  does/does  not  know  and  be  able  to  identify  faulty  reasoning. 
A  soldier  given  the  responsibility  to  conduct  an  AAR  has  the  responsibility  to  be 
prepared  (DA,  25- 1 0 1 , 1 990,  and  TC  25-20, 1 993),  just  as  a  professor  must  prepare  for 
course  instruction.  How  thoroughly  the  AAR  leader  prepares  himself  will  dictate  how 
well  he  will  be  able  to  diagnose  bugs. 

The  last  two  components  of  the  control  structure  are  the  agenda  and  set  of  conflict 
resolution  rules  for  adding/deleting  goals  to/from  the  agenda.  These  goals  and  subgoals 
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are  the  same  ones  explained  earlier  in  this  chapter.  The  priority  for  adding  subgoals  to 
the  agenda  depends  upon  the  type  of  bug  that  is  identified.  These  priorities  are: 

1.  Add  subgoals  to  correct  errors  before  omissions.  Collins  foimd  that  errors  in 
reasoning  had  a  greater  and  far  reaching  affect  on  die  students’  subsequent  reasoning  than 
did  the  omission  of  a  factor.  (Collins  &  Stevens,  1983, 275) 

2.  Add  subgoals  to  correct  prior  steps  before  later  steps.  Barring  a  sequence 
determined  by  the  students’  responses,  the  subgoals  should  seek  to  correct  deficiencies  in 
a  rational  (logical)  order.  (Collins  &  Stevens,  1983, 275) 

3.  Add  subgoals  to  correct  shorter  fixes  before  longer  fixes.  Shorter  fixes  are 
prioritized  over  longer  fixes  because  they  are  easier  to  complete.  (Collins  &  Stevens, 

1983. 275) 

4.  Add  subgoals  that  pertain  to  more  important  factors  before  less  important 
factors.  Obviously,  more  important  factors  have  a  greater  effect  on  the  dependent 
variable  which  makes  their  interrelationships  a  higher  priority  to  learn.  (Collins  & 
Stevens,  1983, 275) 

Collins’  method  for  dealing  with  multiple  bugs  is  simple.  Given  multiple  bugs  in 
the  student’s  reasoning,  the  teacher  always  selects  the  highest  priority  bug  to  fix.  Once 
this  bug  is  corrected,  he  chooses  the  next  highest  priority  bug.  If,  in  the  midst  of  a 
dialogue  (questioning  strategy),  the  teacher  diagnoses  a  higher  priority  bug,  he  interrupts 
(changes  the  subgoal  and  selects  a  new  strategy)  to  correct  that  bug.  (Collins  &  Stevens, 

1983. 276)  However,  Collins  does  not  give  any  guidance  on  prioritizing  bugs  between 
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multiple  students.  In  this  case,  it  seems  that  the  students  themselves  could  help  diagnose 
and  correct  bugs  between  themselves  and  defer  to  the  teacher  at  impasses. 

Is  Inquiry  Theory  Effective? 

There  are  two  remarkable  demonstrations  of  the  effect  that  Inquiry  theory  has  on 
learning;  Valerie  J.  Shute’s  and  Robert  Glaser’s  Smithtown  (Shute  and  Glaser,  1990)  and 
Alan  Lesgold’s,  Gary  Eggan’s,  Sandra  Katz’s,  and  Govinda  Rao’s  Sherlock  (Towne  & 
Munro,  1991,  and  Lesgold  et  al.,  1992)  Both  of  these  are  simulation-based  systems 
designed  to  use  inquiry  strategies  for  domain  specific  knowledge  acquisition.  Each  found 
that  inquiry  skills  dreimatically  increased  the  student’s  ability  to  remember  and  apply  the 
domain  knowledge.  These  systems  are  covered  in  more  detail  in  Appendix  D, 
Implementations  of  Inquiry  Theory. 

Smithtown 

Smithtown  is  an  intelligent  tutoring  system  that  systematically  guides  the 
student’s  discovery  of  microeconomics.  The  system  has  two  goals.  The  first  is  to  aid  the 
student  in  mastering  scientific  inquiry  skills  -  becoming  more  systematic  and  goal 
oriented  in  the  discovery  of  rules  and  theories.  The  second  is  to  impart  microeconomics 
subject  content  to  the  student.  (Shute  and  Glaser,  1990,  51  to  53)  Shute  and  Glaser  fovmd 
that  the  “most  optimum  learner  behaviors  ...  are  systematic,  hypothesis-driven 
activities.”  (Shute  and  Glaser,  1990, 74)  In  other  words,  the  students  who  learned  the 
most,  as  measured  by  pretest  —  posttest  change,  about  microeconomics  learned  and 
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applied  inquiry  skills.  Relative  to  the  pretest,  those  who  did  poorly  on  the  posttest 
resisted  the  guidance  of  the  intelligent  tutor  in  Smithtown  or  were  in  the  control  group 
that  did  not  use  Smithtown.  (Shute  and  Glaser,  1990,  63) 

Shute  and  Glaser  identified  and  addressed  two  problems  when  developing  the 
instruction  tutor  for  induction  reasoning  and  hypothesis  testing.  These  problems  are  that 
“many  learners  can  induce  regularities/pattems  but  do  not  treat  them  as  hypotheses  to  be 
tested”  and  when  they  do  test  a  hypothesis,  many  use  faulty  methods  or  procedures  that 
do  not  guarantee  reasonable  or  relevant  conclusions.  (Shute  and  Glaser,  1990,  52)  These 
problems  were  the  impetus  for  the  goal  to  teach  effective  inquiry  skills. 

Smithtown  is  remarkable  for  instruction  efficiency.  The  performance  difference 
between  students  instructed  by  Smithtown  for  5  hours  and  students  instructed  by 
university  economics  professors  for  1 1  hours  were  not  statistically  significant.  (Shute  and 
Glaser,  1990, 67  and  73)  With  less  than  half  the  instruction  time,  students  using 
Smithtown  performed  just  as  well  as  students  in  an  undergraduate  economics  course  on 
the  same  tests. 

Sherlock 

With  Sherlock,  Lesgold  et  al.  (1992)  establish  a  coached  practice  environment  to 
train  diagnosis  and  repair  of  the  F-15  Manual  Avionics  Test  Station.  The  test  station 
itself  is  used  to  diagnose  failures  of  navigational  equipment  on  the  F-15  aircraft. 
However,  when  the  test  station  fails,  it  must  be  diagnosed  and  repaired  by  a  human 


170 


technician.  The  system’s  goal  is  the  development  and  refinement  of  mental  models  to 
enhance  the  student’s  problem  solving  and  inductive  learning  skills.  (Lesgold  et  al.,  1992, 
51,  and  Towne  &  Munro,  1991, 328) 

Sherlock’s  intelligent  tutor  coaches  the  student’s  problem-solving  performance, 
assesses  student  strengths  and  weaknesses  through  model  tracing,  and  assigns  progression 
problems  based  upon  the  assessments.  (Lesgold  et  al.,  1992, 51)  An  analysis  of  this 
system  reveals  that  the  problem  solving  strategies  are  an  adaptation  of  Inquiry  theory. 

The  students  that  train  with  Sherlock  have  a  basic  working  knowledge  of  electrical 
circuitry.  Thus,  the  students  are  not  novices  and  know  most  of  the  factors  that  may  cause, 
or  combine  to  cause,  a  system  failure.  Sherlock  teaches  the  student  to  efficiently  find  and 
diagnose  the  failure.  In  terms  of  Collins’  Inquiry  theory,  it  accomplishes  this  by 
requiring  the  student  to  form  a  hypothesis  about  the  location  of  the  failure,  test  the 
hypothesis^  and  repeat  the  steps  imtil  the  location  is  determined.  The  student  can  consult 
the  tutor  to  debug  his  hypothesis  by  tracing  consequences  to  a  contradiction,  considering 
alternative  predictions,  varying  cases  systematically,  and  establishing  (selecting)  positive 
and  negative  exemplars.  Lesgold  et  al.  refer  to  this  interaction  with  the  tutor  as 
“reflective  follow-up.”  (Lesgold  et  al.,  1992, 57  through  58)  This  process  is  repeated  for 
diagnosing  the  cause  of  circuit  failure  once  the  location  is  determined.  By  employing 
Inquiry  strategies,  the  student  is  able  to  realize  the  factors  that  indicate  the  location  and 
nature  of  a  failure  in  the  test  station. 
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Lesgold  et  al.  have  developed  a  second  generation  system,  Sherlock  II,  that  has 
not  been  tested  with  students.  However,  the  first  version,  Sherlock,  has  been  tested  for  its 
effect  on  learning.  The  effect  Sherlock  had  on  training  was  determined  by  measuring 
pretest-posttest  change  and  is  even  more  remarkable  than  that  of  Smithtown.  Lesgold  et 
al.  found  that  20  to  25  hours  of  Sherlock  practice  time  produced  average  performance 
improvements  that  were  commensurate  with  the  effects  of  4  years  job  experience. 
(Lesgold  et  al.,  1992, 54)  While  these  results  seem  incredible,  the  learning  gain  from 
Sherlock’s  approach  was  independently  verified  with  32  college  students.  Johnson  et  al. 
(1993)  foimd  that  the  tutorial  group  showed  a  78%  improvement  (in  actual 
troubleshooting  success)  over  the  control  group. 

Although  testing  a  large  number  of  circuit  boards  seems  like  a  mimdane,  time 
consuming  task,  it  is  a  relevant  example  for  an  AAR.  The  location  and  correction  of 
faults  in  the  test  set  requires  the  knowledge  of  a  number  of  different  factors  and  their 
relationships.  While  the  actual  testing  methods  (using  an  oscilloscope  or  multimeter)  are 
simple  tasks,  the  student  must:  1)  consider  what  the  test  station  was  trying  to  do  when  it 
failed;  2)  develop  a  mental  representation  of  the  circuitry  involved  in  the  failed  function; 
and  3)  develop  a  plan  for  testing  that  functional  circuit.  Otherwise,  the  student  is  left 
searching  tens  of  thousands  of  parts  for  the  fault.  (Lesgold  et  al.,  1992, 51  and  52) 
Essentially,  the  student  must  learn  the  cause  and  effect  relationships  between  components 
and  the  troubleshooting  methodology  used  by  the  system  to  narrow  the  choices  for  the 
fault  location.  Then,  the  student  must  develop  an  efficient  circuit  test  plan  to  verify  his 
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hypothesis.  The  better  he  understands  the  factor  relationships,  the  sooner  he  will  isolate 
and  correct  the  problem. 

Like  the  airmen  trained  on  Sherlock,  AAR  participants  have  different  levels  of 
expertise  with  differing  levels  of  realization  about  factor  relationships.  Collectively,  the 
unit  is  not  at  the  novice  level.  What  Sherlock  has  shown  is  that  immense  performance 
improvement  can  be  made  by  focusing  the  students  on  the  problem  and  then  aiding  them 
in  discovery  of  the  factor  relationships  specific  to  that  problem. 

Remarks 

Inquiry  theory  drives  one  to  describe  a  problem  and  its  solution  in  terms  of  the 
factors  and  variables  that  have  an  effect.  These  factor  and  variable  interrelationships  are 
the  declarative  basis  for  productions.  Therefore,  the  discussion  of  these  interrelationships 
amoxmts  to  the  elaboration  that  lays  the  foundation  for  productions  to  be  compiled  and, 
consequently,  skill  learned  or  modified.  In  light  of  ACT-R,  it  follows  that  Inquiry  theory 
is  an  effective  and  efficient  method  of  knowledge  acquisition  and  problem  solving. 

Future  research  planned  for  Sherlock  II  involves  testing  its  effect  on  collaborative 
learning.  (Katz  &  Lesgold,  1993)  This  research  will  implement  Sherlock  II  as  a 
background  tutor,  providing  help  upon  request,  to  problem  solving  groups.  As  such, 
valuable  insights  into  participation  in  a  problem  solving  discussion  may  be  acquired. 

This  effort  is  potentially  the  proof  of  principle  for  the  Inquiry  approaeh  outlined  in  this 
chapter. 
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In  the  meantime,  one  may  wonder  whether  or  not  the  ideas  behind  Smithtown  and 
Sherlock  can  be  generalized  to  a  unit  of  trainees.  Assuming  that  each  trainee  intends  to 
learn  and  improve  performance,  the  short  answer  to  this  question  seems  to  be  “yes.”  The 
reason  Smithtown  and  Sherlock  are  so  successful  is  that  they  employ  a  logical  instruction 
process  that  matches  the  way  humans  process  information  to  learn.  Effectual  facts  are 
related  to  the  causal  facts  at  a  level  commensurate  with  the  individual’s  skill.  Although 
there  are  many  levels  of  skill  within  the  military  group,  each  has  the  goal  of 
understanding  the  knowledge  at  his  skill  level  and  within  the  context  of  the  collective 
skill  level  of  the  group.  Each  trainee  must  understand  how  what  he  knows  and  does  fits 
in  with  the  group’s  actions  and  interactions.  Obviously,  if  the  assumption  that 
individuals  are  training  to  learn  is  invalid,  then,  at  the  very  least,  the  result  will  be  less 
than  full  participation  in  the  AAR  process.  The  nonconformist  trainees  will  not  learn  and 
the  unit  will  not  benefit  from  their  perspective. 

The  nonconformist  trainee  will  impact  on  the  unit’s  learning  synergy.  Each 
member  of  the  group  benefits  from  multiple  perspectives  that  enhance  the  elaboration  of 
learning  points.  Because  someone  of  comparable  skill  and  knowledge  can  relate  it  to 
what  the  individual  already  knows,  the  learning  point  can  be  explained  to  each  team 
member  more  efficiently.  By  making  each  responsible  for  the  other’s  understanding  of  a 
learning  point,  this  last  point  is  a  good  argument  for  increasing  participation  in  die  AAR 
discussion.  This  idea  was  confirmed  by  Hollenbeck  et  al.’s  (1995)  group  decision 
making  research  and  explained  by  Anderson’s  theory  of  cognition  (1993). 
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An  additional  argviment  for  the  generalizeability  of  Smithtown  and  Sherlock  is  the 
nature  of  the  military  unit’s  mission  and  organizational  structure.  In  this  structure,  the 
unit  leadership  is  responsible  for  leading  the  unit  toward  a  single  goal  in  the  AAR: 
performance  improvement.  The  hierarchical  structure  combined  with  the  fact  that  a  small 
unit’s  primary  responsibility  is  to  improve  their  own  performance  are  the  motivations  to 
collectively  learn.  Army  imits  do  not  train  for  any  other  reason;  their  readiness  to  go  to 
war  is  dependent  upon  how  well  they  perform  in  training.  Thus,  the  group  will  work 
together  to  achieve  that  goal  under  the  hierarchy  of  supervision.  It  follows  then  that  the 
unit  can  collectively  employ  Inquiry  theory,  or  any  other  method  of  performance 
improvement,  as  long  as  their  leadership  chooses  to  accept  it.  Research  conducted  by  the 
Naval  Air  Warfare  Training  Systems  Division  (NAWCTSD)  supports  this  position. 

Using  a  structured  AAR  method  called  “guided  team  self-correction”  in  an 
operational  setting,  Smith-Jentsch  et  al.  (1996)  found  that  collective  performance  was 
significantly  enhanced  with  respect  to  an  unguided,  unstructured  method.  In  a  laboratory 
setting,  the  guided  team  self-correction  method  improved  leaning  when  compared  to 
teams  that  were  debriefed  with  a  critique  (i.e.,  one-way  communication  from 
evaluator/trainer  to  a  team).  (Tannenbaum  et  al.,  in  press;  Smith-Jentsch  et  al.,  1996;  and 
Smith-Jentsch,  1996a)  Both  of  these  findings  support  the  hypothesis  that  the  Inquiry 
approach  can  be  generalized  to  unit  learning  (as  indicated  by  performance  improvement). 

Guided  team  self-correction  is  an  AAR  technique  specifically  designed  for 
shipboard  combat  control  operator  teams.  The  technique  is  task-based  in  that  it 
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emphasizes  what  (behaviors)  the  team  must  focus  assessment  on  in  order  to  improve. 
(Smith- Jentsch,  1996a  and  b)  The  cause-effect  performance  factors  are  identified  and 
stable  with  respect  to  the  shipboard  (immediate)  environment.  On  the  other  hand,  the 
Inquiry  approach  provides  trainees  a  how  to  method  for  performance  diagnosis  when  the 
causes  are  unknown  and/or  effects  unclear.  The  difference  between  the  Army  and  Navy 
performance  situations  is  that  Army  units  operate  in  environments  that  are  more  dynamic. 
The  Navy  team’s  environment,  the  ship’s  combat  control  center,  is  stable  and  therefore 
not  a  source  of  performance  variation.  Notwithstanding,  the  main  point  is  that  guided 
discovery  techniques  can  improve  both  collective  and  individual  performance. 


Effectiveness  of  the  Proposed  Approach 
Participation 

As  shown  in  Chapters  II  and  III,  participation  is  a  key  and  elusive  ingredient  of  an 
AAR.  The  proposed  approach  assigns  the  AAR  leader  the  responsibility  to  prompt, 
monitor,  and  reward  participation  during  the  AAR.  In  actuality,  this  was  one  of  the 
original  duties  of  the  AAR  leader  when  the  AAR  was  being  developed  as  a  training 
feedback  concept.  (Scott,  1983,  Bosley  et  al.,  1979,  Shriver  et  al.,  1975,  and  USAIS, 
1974)  These  same  sources  point  out  that  once  the  soldiers  are  participating,  momentum 
is  gained  and  any  discussion  inhibitions  are  lost.  Observation  of  the  4  participatory, 
discussion  focused  AARs  and  subject  matter  experience  confirm  this  observation. 
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The  AAR  leader  must  also  provide  feedback  to  individuals  and  the  group  on  their 
participation  in  the  discussion.  (Argyris,  1994,  and  Smith  &  Right,  1959)  The  study  of 
17  AARs  in  Chapter  III  confirmed  that  elements  of  participation  increased  with  feedback. 
Hence,  if  the  AAR  leader  can  verbally  laud  a  soldier  for  contributing  to  the  discussion,  he 
and  the  other  unit  members  will  be  more  likely  to  participate.  This  feedback 
responsibility  augments  Collins’  Inquiry  strategy  of  questioning  authority.  This  is  the 
strategy  that  requires  the  trainees  to  be  self-reliant  in  learning.  They  do  not  rely  upon  the 
AAR  leader  or  senior  trainer  for  answers  but  must  find  them  on  their  own.  This  frees  the 
AAR  leader  to  identify  “bugs”  and  facilitate  articulation  and  elaboration  by  suggesting 
the  use  of  the  terrain  model,  chalk  board,  or  reference  manuals. 

The  AAR  leader  can  also  physically  involve  participants  in  the  AAR.  He 
accomplishes  this  by  having  different  soldiers  draw  concept  sketches  on  a  blackboard, 
move  personnel  or  vehicle  models  on  a  terrain  board,  etc.  This  gets  the  soldiers  out  of 
their  seats  where  it  is  too  comfortable  to  sit  passively.  Participation  by  physical 
movement  can  be  a  first  step  toward  influencing  a  person  to  submit  his  ideas  verbally. 

Whiteboards  and  terrain  models  can  also  help  the  participant  articulate  his  point. 
Many  combat  arms  soldiers  cannot  verbally  communicate  effectively.  These  aids  can 
help  soldiers  organize  their  thoughts  and  explain  relationships  to  one  another;  thus,  they 
help  the  participants  elaborate  the  learning  points. 

Hence,  it  is  the  AAR  leader’s  responsibility  to  actively  monitor  and  facilitate 
participation.  He  accomplishes  this  by  explicitly  setting  participation  as  an  objective  of 
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the  AAR  and  actively  attending  to  that  objective.  He  must  be  sensitive  to  the  trainees’ 
inhibition  to  submit  observations  and  ideas  for  discussion.  Additionally,  he  must  beware 
of  inaccurate  or  incomplete  articulations  and  intervene  to  prompt  correction  or 
completion  of  statements.  Inaccurate  statements  can  easily  be  corrected  by  asking  the 
trainee  group,  or  a  specific  individual,  to  verify  or  correct  the  statement.  The  AAR  leader 
must  ensure  that  all  the  participants  who  were  involved  during  the  training  exercise 
submit  their  experience.  Those  who  have  trouble  explaining  what  they  saw  and  why  they 
acted  are  encouraged  to  use  articulation  tools  such  as  the  terrain  board  or  whiteboard. 

The  AAR  leader  also  must  encourage  dissenting  views  so  that  participants  are 
forced  to  critique  and  defend  their  arguments.  Once  their  arguments  are  on  the  table  for 
critique,  those  participants  have  a  vested  interest  in  participating  in  the  discussion.  If  a 
participant’s  argument  is  proven  faulty  by  the  group,  the  AAR  leader  must  minimize  the 
individual’s  embarrassment  by  rewarding  him  for  offering  the  comments  and  encouraging 
his  continued  input.  He  can  do  this  by  pointing  out  the  components  of  the  argument  that 
have  merit  despite  the  overall  failure  when  the  components  are  combined.  This  draws  the 
groups  attention  to  what  is  said  and  not  who  said  it.  If  the  AAR  leader  can  influence  the 
group  to  collectively  critique  the  ideas  in  a  rational  manner  without  respect  to  who 
submitted  the  idea,  then  individual  inhibitions  will  be  perceived  as  less  of  an  obstacle  to 
participation.  (Argyris,  1994) 

The  AAR  leader  has  one  irrefutable  argument  that  he  can  use  to  motivate 
participation.  As  soldiers  in  the  US  Army,  it  is  their  duty  to  conduct  and  participate  in 
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training.  Since  the  AAR  is  a  part  of  that  training,  then  tiiey  are  also  duty  bound  to 
participate  in  the  AAR.  The  soldiers  may  need  to  be  reminded  that  participation  means 
active  contribution  and  not  passive  reception.  This  argument  combined  with  a  reminder 
that  the  AAR  is  for  dieir  benefit  -  to  improve  their  performance  and  nobody  else’s  -  was 
a  substantial  influence  toward  open  discussion  in  the  AAR  study.  (Chapter  III)  The  AAR 
leader  must  de-emphasize  his  own  role  and  emphasize  the  trainees’  role.  This  is 
fimdamental  to  establishing  an  environment  in  which  the  trainees  take  ownership  of  the 
discussion  and  the  ideas  that  are  developed  during  the  discussion. 


Discussion  Focus 

Smithtown  showed  that  the  application  of  Inquiry  theory  to  a  group  of  students 
increases  the  effectiveness  and  efficiency  of  learning  the  problem  solving  methodology  as 
well  as  the  subject  content.  If  done  well.  Inquiry  teaching  fosters  multiple  hypotheses 
and  argumentation  among  the  group.  (Collins,  1987,  186)  Hence,  the  implementation  of 
the  theory  will  increase  participation  and  the  number  of  points  of  view  volunteered  during 
the  AAR.  The  theory’s  application  will  also  increase  the  elaboration  and  articulation 
efforts  of  a  participant  as  he  submits  and  defends  his  views  and  hypotheses.  This  effort 
translates  into  individual  improvement  as  each  learns  what  and  how  cause  and  effect 
relationships  affect  performance.  The  unit  improves  as  the  shared  mental  model  becomes 
more  clearly  defined  during  the  discussion. 
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The  focus  of  the  discussion  is  critical  to  AAR  effectiveness.  To  correct  a 
problem,  the  problem  itself  must  be  studied.  Beginning  the  AAR  discussion  with  the 
BDA  was  highlighted  as  a  time  efficient  method  for  participants  to  identify  a  problem 
topic.  (Chapter  II,  27,  Chapter  III,  AAR  Survey)  If  the  process  is  followed,  the  AAR 
participants  are  guided  to  meet  the  doctrinal  purpose  and  objectives  of  an  AAR 
discussion.  Specifically,  this  is  what  happened,  why  it  happened,  and  a  comparison  of 
proposed  solutions  of  how  to  do  it  better.  Inquiry  theory  is  a  time  efficient  and  thorough 
method  of  focusing  discussion  on  a  problem  topic.  As  Word  pointed  out,  the  more  time 
and  effort  spent  on  clarifying  why  something  happened,  the  easier  it  becomes  to  realize 
how  to  correct  the  problem  and,  subsequently,  generalize  the  solution  to  another  situation. 
(Word,  1987, 34) 

Using  the  BDA  from  a  training  exercise  as  a  point  of  departure,  a  casualty  at  a 
specific  point  of  the  battle,  for  example,  can  be  selected  as  the  dependent  variable.  The 
participants  then  use  Inquiry  strategies  to  systematically  identify  the  factors  that  caused 
the  casualty  and  the  variables  that  influence  those  factors.^’  Given  a  specific  outcome  of 
a  training  exercise,  the  questions  that  the  participants  need  to  address,  in  order,  are: 

1.  What  are  the  factors  that  caused  this  outcome? 

2.  What  are  the  variables  that  affect  these  factors? 

3.  How  do  they  interrelate  to  cause  the  outcome? 


In  this  case,  I  am  defining  a  “factor”  as  a  set  of  variables  that  can  be  logically  grouped.  An  example  of 
this  could  be  the  METT-T  (mission,  enemy,  terrain,  troops,  and  time)  conditions  of  a  battle.  Each  consists 
of  a  number  of  variables,  i.e.  terrain:  weather,  vegetation,  relief,  visibility,  etc. 
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4.  Which  factor(s)/variable(s)  can  be  adjusted  to  alter  the  outcome? 

5.  Can  the  desired  outcome  be  achieved  by  controlling  these  factor(s)/variable(s)? 
Answering  these  generic  questions  will  determine  why  something  happened  and  how  to 
do  it  differently. 

The  important  point  at  this  juncture  is  to  be  thorough.  All  of  the  factors  that 
affected  the  outcome  must  be  identified.  Repeatedly  applying  these  questions  to  a 
situation  and  outcome  will  reveal  the  root  cause(s)  of  the  outcome  and  the  necessary 
interventions  to  alter  the  outcome  to  the  desired  state.  These  interventions  are  the 
adjustments  to  factors  that  the  participants  hypothesize  will  affect  the  outcome.  They  are 
forming  hypotheses  and  varying  cases  systematically.  Obviously,  the  unit  can  only  affect 
the  factors  and  variables  that  they  control. 

The  next  step  is  to  test  the  hypothesis  imder  the  training  conditions  in  which  the 
unit  performed  to  prove  that  the  desired  outcome  can  be  achieved.  If  the  outcome  is  not 
affected  or  the  desired  outcome  is  not  achieved,  the  xmit  has  two  courses  of  action.  Either 
the  unit  can  trace  the  consequences  to  a  contradiction  and  correct  the  misconeeived 
interrelationship,  or  it  must  begin  the  analysis  to  identify  the  factors,  variables,  and  their 
interrelationships  over  again  and  re-hypothesize  the  factors  and  variables  that  will  affect 
the  outcome. 

If  the  desired  outcome  is  achieved,  then  systematic  testing  must  be  conducted  to 
determine  over  what  range  of  conditions  the  hypothesis  holds  for.  This  inquiry  strategy  is 
called  generating  hypothetical  cases  and  causes  the  participants  to  generalize  the  factor 
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and  variable  relationships  to  differing  situations.  Essentially,  each  of  the  hypothetical 
cases  becomes  a  new  hypothesis  that  must  be  tested  and  proven  true  or  imtrue.  The 
hypotheses,  set  of  relationships  that  affect  and  achieve  the  desired  outcome,  are  the 
answers  to  the  question  of  how  the  unit  can  perform  to  achieve  the  specified  performance 
measure/outcome  -  improve  performance.  The  factors  that  are  common  to  each 
hypothesis  that  is  proven  true  are  the  conditions  and  actions  that  must  be  replicated  and 
performed  in  future  training  to  achieve  the  improvement. 

The  end  result  of  the  Inquiry  approach  is  that  the  participants  know:  1)  what 
happened;  2)  why  it  happened  with  respect  to  the  battlefield  conditions,  other  actions, 
and  doctrinal  principles;  and  3)  given  different  battlefield  conditions,  how  to  manipulate 
different  resources  imder  their  control  to  achieve  a  desired  effect.  They  will  have 
articulated  and  elaborated  the  future  training  task,  conditions,  and  performance  standards 
needed  to  reinforce  the  behavior  required  to  achieve  the  desired  effect.  This  is  the  future 
training  plan. 

Via  the  Inquiry  process,  the  participants  will  gain  a  greater  understanding  of  the 
doctrinal  factors,  variables,  and  interrelationships  that  affect  unit  performance  through 
discussion  amongst  themselves.  Inquiry  theory  focuses  the  discussion  on  what  is 
important  and  guides  it  to  the  end  result  of  how  to  do  what  is  important  better.  The  use  of 
Inquiry  strategies  allows  one  to  construct,  reinforce,  and  delete  production  links 
effectively  and  efficiently,  in  real  time.  The  result  is  improved  performance. 
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In  other  AAR  systems,  the  focus  and  guidance  of  the  discussion  were  the  AAR 
leader’s  sole  responsibility.  Furthermore,  in  most  of  the  AARs  surveyed,  the  AAR  leader 
presumed  the  content  of  the  discussion.  This  is  why  AAR  leaders  tend  to  lecture  and 
dominate  the  AAR  discussion.  Inquiry  theory  automatically  guides  and  focuses  the 
discussion,  leaving  the  AAR  leader  to  concentrate  on  reducing  the  inhibitions  to 
discussion  participation. 

The  AAR  participants  learn  the  Inquiry  strategies  and  approach  through 
repetition.  It  is  intuitive  that  repeated  exposure  to  a  specific  approach  teaches  the  trainee 
that  approach.  Anderson’s  ACT-R  theory  of  cognition  verifies  this  idea  -  the  process 
becomes  proceduralized.  (Anderson,  1993).  For  example,  someone  learning  how  to 
check  the  oil  level  in  an  engine  block  repeatedly  pulls  the  dipstick,  wipes  it  clean, 
reinserts  it  into  the  crankcase,  withdraws  the  dipstick  again,  and  then  reads  the  level. 
After  checking  the  oil  levels  in  10  cars,  he  cannot  remember  what  the  levels  were,  but  he 
can  remember  the  process  that  was  used. 

Medical  diagnosis  training  also  provides  a  common  example  of  repetitive  practice 
and  elaboration  to  learn  a  complex  process.  Interns  must  spend  a  year  diagnosing  patient 
after  patient  imder  rigorous  supervision  to  learn  the  process  of  accurate  diagnosis. 
(Bruffee,  1993, 24  through  25)  The  routine  use  of  Inquiry  strategies  to  diagnose 
performance  is  a  similar  process.  Units  and  individuals  will  learn  that  process  by 
conducting  a  number  of  standardized  AARs  over  an  extended  period  of  time. 
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Time 


As  stated  before,  the  application  of  Inquiry  theory  is  a  time  efficient  method  of 
solving  problems.  With  respect  to  ACT-R,  Inquiry  theory  identifies  the  concatenation  of 
productions  that  define  the  problem  skill.  (Anderson,  1993, 33  through  34)  The  use  of 
this  methodology  ensures  that  the  discussion  is  focused  upon  what  is  relevant  -  the 
factors  and  variables  that  affect  unit  performance.  Therefore,  if  the  Inquiry  approach  is 
followed  and  the  discussion  topic  is  appropriate,  the  AAR  time  will  not  be  used  in  the 
aimless  exploration  of  irrelevant  data. 

Inquiry  theory  also  supports  the  quick  preparation  of  the  AAR  simply  by  utilizing 
the  participant’s  knowledge  of  the  exercise.  The  exercise  information  does  not  need  to  be 
prepared  and  presented  in  a  specific  format.  If  the  AAR  is  conducted  immediately  after 
the  training  exercise,  then  the  trainees  can  recall  the  information  at  relevant  points  during 
the  inquiry  process.  An  alternative  is  that  the  simulation  training  system’s  data  collection 
system  responds  to  real  time  requests  to  present  the  relevant  performance  information. 
This  obviates  the  need  to  construct  a  pre-formatted  presentation.  The  information  that 
must  be  provided  to  begin  the  AAR  is  the  data  associated  with  the  performance  standards 
of  the  training  tasks  -  the  performance  results.  These  are  the  collective  task  standards 
that  are  specified  by  the  objectives  of  the  training  plan. 

The  system  can  work  equally  well  for  live  simulations.  Given  that  the  trainees 
can  act  as  the  repository  for  exercise  information,  the  AAR  leader  needs  only  the 
summary  of  performance  outcomes  relevant  to  the  training  objectives  and  doctrinal 
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performance  standards  and  training  in  the  Inquiry  problem  solving  approach.  The  Inquiry 
strategies  are  a  collection  of  topic  focused,  open-ended  and  leading  questions.  The 
Inquiry  approach  will  efficiently  guide  the  AAR  participants  through  their  own  memory 
of  the  exercise  and  knowledge  of  doctrinal  relationships. 

The  AAR  leader’s  skill  at  employing  the  Inquiry  strategies  will  determine  how 
much  effort  his  is  able  to  dedicate  to  discussion  participation.  As  found  in  the  survey,  the 
AAR  leader  who  was  unskilled  in  the  questioning  technique  spent  most  of  his  time  and 
effort  in  the  formulation  of  questions.  Consequently,  he  had  little  time  to  give 
participation  feedback. 


Learning  Reinforcement 

Once  the  interrelationships  between  a  number  of  factors  and  variables  are 
understood  and  the  conditions  to  which  they  apply  are  known,  the  key  to  learning  is 
repeated  elaboration  and  practice.  This  is  intuitive  -  it  makes  sense.  With  ACT-R, 
Anderson  convincingly  demonstrated  that  learning  is  the  conversion  of  the  declarative 
factor  and  variable  interrelationships  to  production  rules  through  the  generalization  of 
previous  experience.  Trainees  must  relate  the  new  declarative  facts  and  productions  to 
what  they  already  know.  The  key  to  a  group  learning  complex  collective  tasks  is  that  the 
group’s  common  experience  is  iteratively  generalized  and  practiced. 

The  task  to  be  learned  becomes  complex  when  people  must  rely  on  each  other  to 
perceive  environmental  cues  and  then  perform  the  appropriate  portions  of  the  overall  task. 
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This  complexity  makes  the  task  difficult  to  collectively  accomplish  as  a  team.  The  task 
becomes  even  more  complex  when  team  members:  are  dependent  upon  each  other,  are 
allowed  to  communicate  only  intermittently,  receive  too  many  environmental  cues  to 
process,  and  the  consequences  of  action  or  inaction  are  both  emotionally  significant  and 
life  threatening.  At  the  small  unit  level  in  this  environment,  living  or  dying  depends  upon 
the  group  of  soldiers  performing  a  specific  sequence  of  actions.  First,  they  must 
imderstand  and  work  toward  a  common  goal;  usually  the  destruction  of  the  enemy  force. 
Second,  they  must  individually  and  collectively  interpret  the  battlefield  cues  so  that  they 
perceive  the  same  meanings  and  significance;  this  is  the  hardest  step  of  all.  Third,  given 
the  perceived  conditions  (cues),  they  must  individually  and  collectively  select  the  same 
action  or  reaction  in  order  to  achieve  the  common  goal.  Fourdi,  they  must  execute  that 
action  in  a  uniform  manner  such  that  everyone  imderstands  what  already  has,  needs,  and 
remains  to  be  done.  The  trainees  must  construct  a  shared  mental  model. 

The  reinforcement  of  acquired  knowledge  in  the  AAR  is  needed  to  both  build  and 
learn  the  structure  of  a  shared  mental  model.  In  the  proposed  AAR  approach,  the 
reinforcement  occurs  as  collective  mental  practice  and  hypothesis  testing.  After  the 
factors  and  variables  that  affected  performance  during  training  are  defined,  the  trainees 
generalize  these  interrelationships  to  different  situations.  To  do  this,  the  trainees  must 
mentally  vary  the  independent  factors  and  variables  under  their  control  to  achieve  the 
desired  effect  in  the  given  environment  -  this  amoimts  to  mental  rehearsal.  The  ability  to 
test  the  generalization  with  the  testing  component  signifies  the  generalized  conclusions. 
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In  this  exercise  of  hypothesis  formulation  and  testing,  each  trainee  can  input 
considerations  for  both.  This  participation  helps  to  ensure  the  hypothesis  formulation  and 
testing  are  necessarily  thorough. 

There  are  a  number  of  tools  available  to  facilitate  hypothesis  formulation  and 
testing.  Whiteboards  (or  large  sketch  pads)  are  common  in  CTC  and  homestation 
training  AARs.  (CTC  AAR  survey  and  subject  matter  experience)  Word  (1987)  and 
Scott  (1987)  emphasized  the  use  of  a  scaled  terrain  model.  Personal  subject  matter 
experience  confirms  that  the  terrain  model  greatly  enhances  elaboration  and  articulation 
of  the  considerations  needed  for  hypothesis  formulation  and  testing.  In  Appendix  E, 
Proposed  Design,  a  3D  holographic  projection  system  and  a  constructive  simulation  are 
suggested  for  terrain  model  creation  and  hypothesis  testing  respectively Thus,  the 
hypothesis  test  results’  significance  is  further  reinforced  by  the  holographic  display  which 
enhances  the  trainees’  visualization  ability.  The  trainees  realize  the  required  actions  for 
performance  improvement  when  presented  different  battlefield  cues  by  physically  seeing 
them. 

A  constructive  simulation  could  facilitate  mental  practice.  The  replay  of 
generalized  hypotheses  would  allow  the  trainees  to  elaborate  on  the  actions  necessary  to 
achieve  a  specific  goal.  This  elaboration  is  important  to  relating  the  hypothesized  actions 
to  tasks  and  subtasks  the  trainees  already  know  and  understand.  In  terms  of  Anderson’s 
ACT-R  theory  of  cognition  (1993),  this  elaboration  is  strengthening  the  connection 
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strength  between  the  productions  that  were  derived  in  the  AAR  and  the  productions  that 
already  existed  from  previous  training  and  experience.  The  stronger  the  tie  is  to  what  is 
already  known,  the  more  efficient  the  recall  is  of  what  was  just  learned.  (Anderson,  1993) 
The  simulation  could  enhance  the  trainees’  visualization  as  they  iteratively  solve  sub¬ 
problems  by  analogy.  Of  course,  combining  this  mental  rehearsal  with  physical  rehearsal 
will  strengthen  the  connection  so  that  recall  is  even  more  efficient.  More  is  said  on  how  a 
constructive  simulation  could  be  used  in  this  manner  in  Appendix  E,  Proposed  Design. 

An  important  benefit  of  this  approach,  besides  a  common  mental  model  of 
responses  to  the  battlefield  environment,  is  that  the  trainees  understand  what  mission, 
enemy,  terrain,  troop,  and  time  conditions  are  needed  to  train  the  umt  so  that  a  specific 
outcome  is  achieved.  Plans  for  future  training  are  developed  and  tested  during 
generalization.  The  trainees  realize  what  battlefield  cues  are  absolutely  necessary  for  the 
training  of  a  specific  task.  This  is  important  because  training  resources  dictate  what 
battlefield  cues  can  be  realistically  replicated  for  training. 

If  the  unit  knows  its  resource  needs  then  it  can  request  those  resources  and  plan 
training  accordingly.  Often,  what  occurs  instead  is  that  units  know  what  tasks  must  be 
trained  but  do  not  consider  what  resources  are  needed.  Unfortunately,  they  train  the  task 
with  the  available  resources  which  is  often  a  hit  or  miss  undertaking.  Quite  simply,  either 
the  available  resources  combine  to  create  the  necessary  cues  to  elicit  the  actions  that  need 
practice,  or  they  do  not.  The  proposed  system  will  highlight  the  needed  resources  for  the 

^  A  good  example  of  the  holographic  projection  system  envisioned  is  the  Laser-based,  3D  Volumetric 
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corrective  training  of  specific  weaknesses  and  the  sustainment  training  of  specific 
strengths.  With  this  knowledge,  units  can  either  acquire  the  needed  resources  for  training 
or  train  on  tasks  that  the  available  resources  will  support,  but  they  will  not  unknowingly 
waste  time  and  effort  trying  to  achieve  a  specific  performance  effect  by  training  tasks  that 
cannot  be  properly  resourced. 


Display  System.  (Soltan  et  al.,  1995) 
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CHAPTER  V 


FINDINGS,  LESSONS  LEARNED  AND  SUGGESTED  FUTURE  RESEARCH 


" . .  .the  far  object  of  a  training  system  is  to  prepare  the  combat  officer 
mentally  so  that  he  can  cope  wititi  the  unusual  and  ihe  unexpected  as  if  it 
were  the  altogether  normal  and  give  him  poise  in  a  situation  where  all 
else  is  in  disequilibrium. .  .the  beginning  lies  in  a  system  of  schooling 
which  puts  the  emphasis  on  teaching  soldiers  how  to  think  rather  than 
what  to  think  even  though  such  a  revolutionary  idea  would  put  the  army 
somewhat  ahead  of  our  civilian  education." 

-  S.  L.  A  Marshall 
Men  Against  Fire.  1947 


This  research  considered  the  AAR  in  light  of  Army  doctrine,  AAR  research, 
performance  feedback  (behavioral  psychology),  and  cognitive  learning  theory  literature  to 
determine  the  elements  of  an  effective  AAR.  Given  these  elements,  a  methodology  was 
then  developed  to  assess  effectiveness  in  an  actual  AAR.  An  analysis  of  17  CTC  AAR 
cases  provided  insight  to  some  deficiencies  with  respect  to  the  elements  of  effectiveness. 
This  study  and  a  review  of  current  simulation  training  AAR  support  systems  are  used  as 
the  basis  to  suggest  the  AAR’s  current  state  and  potential  with  respect  to  effectiveness.  A 
derivation  of  Inquiry  theory,  an  instructional  strategy  of  meta-cognitive  skills,  was 
presented  as  an  approach  to  overcome  these  deficiencies  and  achieve  effectiveness  in  an 
AAR. 
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Findings 


Learning  does  not  require  detailed  and  laborious  erudition.  Humans  learn  through 
the  systematic  acquisition  of  knowledge  by  analogy,  this  is  the  dynamic  combination  of 
declarative  and  procedural  knowledge  with  respect  to  what  is  already  known  and  a 
problem  goal.  To  collectively  learn,  the  group  must  develop  a  common  rmderstanding  of 
the  problem  space  and  solution  -  a  common  mental  model  of  what  is  wrong  with 
performance  and  how  to  improve  it.  The  requirements  for  this  development  are: 

1 .  A  knowledge-based  domain; 

2.  Objective  performance  feedback  (based  upon  what  actually  happened)  that 

supports  the  doctrinal  task  based  training  system; 

3.  An  AAR  leader  who  guides  and  focuses  discussion  and  provides  participatory 

feedback  to  the  group; 

4.  Articulation  tools  to  aid  in  defining  the  problem  space; 

5.  A  logical  problem  solving  methodology  that  requires  problem  definition, 

cause-effect  factor  analysis,  and  solution  testing; 

6.  Elaboration  tools  to  aid  visualization  and  mental  practice;  and 

7.  Training  tools  to  aid  physical  practice. 

An  effective  AAR  system  must:  promote,  if  not  require,  participation  by  all  of  the 
trainees;  focus  discussion  on  the  learning  points  that  affect  performance;  reinforce  the 
learning  points  through  hands-on  application  of  problem  solutions;  and  allow  timely 
conduct  of  the  AAR.  These  elements  are  interdependent.  The  key  to  achieving  the  first. 
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second,  and  fourth  elements  of  effectiveness  is  the  employment  of  a  structured  problem 
solving  method  combined  with  a  discussion  facilitator  who  provides  participation 
feedback  to  the  group.  The  discussion  is  where  the  learning  that  affects  (improves) 
performance  takes  place.  The  Inquiry  approach  is  the  structured  acquisition  of  new  facts 
in  a  logical  sequence  so  that  the  right  associations  can  be  made  and  productions  formed. 
In  effect,  the  approach  guides  the  efficient  and  effective  construction,  reinforcement,  and 
modification  of  productions.  The  application  of  Inquiry  theory  translates  into  efficient 
learning. 

The  third  element  of  effectiveness,  learning  reinforcement,  is  not  considered  by 
the  doctrinal  AAR  process  nor  simulation  training  AAR  subsystems.  Learning 
reinforcement  allows  the  imit  and  individuals  to  proceduralize  the  learning  point  through 
verbal  elaboration  and  physical  application.  In  other  words,  trainees  learn  through 
mental  and  physical  practice.  AAR  participants  need  tools  that  facilitate  self-elaboration 
and  allow  timely  practice  of  the  learning  points  discussed.  Hence,  this  component  also 
has  a  time  sensitive  element.  Trainees  must  practice  what  they  learn  as  soon  as  possible 
before  the  declarative  knowledge  representation  decays. 

A  systematic  problem  solving  approach  combined  with  elaboration  increases 
learning  and  performance.  It  is  achieved  with  Inquiry  theory  by  questioning  and 
explaining  the  learning  points  of  the  AAR.  Elaboration  can  be  pursued  most  efficiently 
as  a  group  discussion  since  multiple  perspectives  are  offered  and  articulated.  The 
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multiple  perspectives  help  each  individual  realize  the  learning  point  by  relating  it  to  what 
he  already  knows. 

Self-elaboration  is  a  method  to  problem  solve  by  analogy.  ACT-R  provides  a 
clear  and  plausible  theoretical  basis  for  the  premise  that  elaboration  is  needed  to  learn,  as 
demonstrated  by  both  Smithtown  and  Sherlock.  These  systems  employ  versions  of 
Inquiry  theory  to  elicit  elaboration  of  learning  points  from  the  student,  keep  the  student 
focused  on  the  learning  point  until  he  understands  it,  and  then  require  him  to  apply  or 
demonstrate  the  learning  point.  The  mechanism  that  allows  the  student  to  understand  the 
point  is  iterative  elaboration  of  experience.  In  the  end,  self-elaboration  becomes 
synonymous  with  mental  practice. 

An  AAR,  or  performance  feedback,  system  needs  components  and  processes  that 
prompt  and  focus  discussion.  These  components  serve  as  a  medium  for  discussion.  They 
help  the  trainees  learn  by  articulating  and  elaborating  concepts,  rules,  and  theories  to  a 
point  of  common  understanding.  The  AAR  system  components  must  also  provide  for 
learning  reinforcement  of  the  discussion  points.  This  provides  the  unit  the  opportunity  to 
physically  implement  or  practice  the  learning  points  they  have  elaborated  and  articulated; 
thus,  imbedding  the  points  in  their  experience  and  specifying  the  conditions  for  future 
training. 

The  final  requirement  is  that  the  approach  must  not  prolong  AAR  preparation  but 
allow  timely  learning  reinforcement.  For  platoon  and  company  level  training  exercises, 
the  AAR  must  commence  within  an  hour  of  the  termination  of  training.  The  duration  of 
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the  AAR  is  usually  limited  by  time  because  of  training  system  co-use  with  other  units. 
Therefore,  the  AAR’s  duration  must  be  dedicated  to  efficiently  learning  rule  and  theory 
relationships  that  affect  performance  and  not  by  lectures  of  what  and  what  not  to  do. 
Hence,  the  AAR  system  must  get  participants  into  the  mode  of  discussion  as  quickly  as 
possible. 

In  summary,  the  goal  of  the  proposed  AAR  approach  has  both  short  and  long  term 
orientations.  For  the  short  term,  the  objective  is  to  increase  the  trainees’  level  of 
expertise  and  improve  performance  with  respect  to  doctrinal  standards.  The  long  term 
objective  is  to  create  a  collaborative  learning  environment  in  which  the  unit  learns  the 
problem  solving  skills  necessary  to  efficiently  diagnose  and  critique  performance.  This 
approach  supports  the  overall  goal  of  the  AAR:  to  improve  individual  and  unit 
performance. 


Research  Limitations 

The  methodology  to  measure  effectiveness  of  an  AAR  does  and  will  not  indicate 
how  much  the  unit  learned.  It  only  indicates  whether  or  not  the  conditions  for  more 
efficient  learning  are  created.  Although  the  survey  is  only  an  indicator  of  the  AAR 
system  at  the  CTCs,  it  does  strongly  confirm  my  subject  matter  experience;  AARs  are 
generally  non-participatory  and  do  not  focus  on  how  a  solution  can  be  implemented  so 
that  the  unit  learns.  Even  though  the  CTC  AAR  system  is  the  example  all  follow,  units 
vary  in  the  implementation  methods  of  that  example.  Hence,  a  larger  sample  (n)  is 
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needed  to  make  a  conclusive  assessment  about  how  effective  AARs  are  across  the  Army. 
There  will  most  likely  be  an  extreme  variation  between  unit  AARs  for  the  simple  reason 
that  doctrinal  AAR  guidance  is  not  a  standard  subject  taught  in  Army  schools.'** 

The  survey  is  an  indication  of  infantry  and  armor  platoon  and  company  AARs. 
Other  unit  types,  field  artillery,  air  defense  artillery,  military  police,  etc. . .,  were  not 
surveyed.  Nor  were  Air  Force,  Navy,  and  civilian  organizations  that  employ  training 
feedback  studied.  The  other  branches  of  the  Army  and  military  services  are  more 
technically  oriented  than  infantry  or  armor  organizations.  Indeed,  the  infantry 
organization  is  the  only  one  whose  most  important  system  is  the  soldier.  The  armor  units 
include  the  tank  crew  (the  people)  as  a  component  of  the  tank  system.  The  point  is  that 
the  technically  oriented  organizations  require  more  technical  and  individualized  feedback. 
(SHERIKON,  1996b)  For  example,  the  pilot  in  the  F- 16  is  most  concerned  with  his 
ability  to  fly  the  airplane,  as  he  should  be.  Notwithstanding  this  technical  nature,  the 
Inquiry  approach  should  accommodate  any  (individual  or  collective)  performance 
oriented,  task  based,  training  system.  Hence,  fire  departments,  police  departments,  and 
emergency  medical  teams  could  all  benefit  from  this  approach  and  subsequent  research. 
Furthermore,  higher  level  military  organizations  such  as  brigade  and  division  staffs  could 
also  employ  the  Inquiry  approach.  That  is,  if  they  are  willing  to  submit  their  actions  to 
scrutiny  in  terms  of  tasks  with  measurable  performance  standards. 


Army  schools  are  officer/NCO  basic  and  advanced  courses,  staff  schools,  and  command  courses. 
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Lessons  Learned 


Problems 

Defining  Participation 

Participation  is  not  described  adequately  in  research  literature.  Conclusive  studies 
have  not  been  conducted  to  show  when  and  how  participation  affects  learning.  More  to 
the  point,  the  factors  that  motivate  a  person  and  group  of  people  to  proactively  participate 
in  a  potentially  embarrassing  discussion  (where,  at  least,  self-esteem  is  at  stake)  have  not 
been  isolated  and  demonstrated  convincingly.  Current  and  future  research  in 
collaborative  learning  should  provide  the  necessary  empirical  evidence. 

It  is,  however,  intuitive  that  participation  positively  affects  learning.  Although 
not  directly  measured  in  the  AAR  study,  participation  seemed  highest  when  practical 
solutions  were  being  discussed.  In  other  words,  trainees  were  motivated  to  solve 
problems.  Conversely,  when  solutions  were  obvious,  irrelevant,  or  not  discussed  at  all, 
trainees  were  less  motivated  to  participate.  Once  shown,  this  evidence  will  provide  the 
basis  for  the  R  &  D  of  simulation  tools  to  directly  increase  participation  in  a  group 
learning  setting. 

Survey  Methodology 

Variability  in  data  collection  was  not  adequately  controlled.  Although  data 
collection  was  practiced  on  two  AARs,  an  analysis  to  measure  the  reliability  of  collection 
was  not  done.  Undoubtedly,  there  is  some  variability  in  data  collection  between  and. 
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perhaps,  within  an  AAR.  There  are  two  ways  to  easily  reduce  some  of  this  variability. 
One  is  to  watch  the  tapes  twice.  This  would  allow  data  to  be  recorded  that  had  been 
missed  the  first  time.  The  second  is  to  utilize  multiple  recorders  for  the  AAR  leader  and 
groups  of  trainees.  This  option  would  ensure  that  a  data  collector  is  observing  only  a 
small  number  of  participants  at  one  time.  In  the  AARs  observed  for  this  study,  an 
observer  could  easily  record  the  data  for  four  participants. 

Multicollinearity  was  a  problem  in  the  regression  analysis.  This  resulted  in  some 
of  the  findings  from  the  trend  analysis  not  being  statistically  verified.  An  initial  solution 
to  this  problem  is  to  transform  the  variables  with  a  logistic  response  fimction.  (Neter  & 
Wasserman,  1974)  This  would  allow  a  smoother  approximation  of  correlated  variables  in 
each  regression  model. 

A  second  possible  solution  is  to  redefine  the  variables  so  that  they  are  not 
correlated.  For  example,  the  imit  attitude  and  unit  leader  action  variables  would  require  a 
detailed  analysis  of  the  unit  leader  -  trainees  relationship  to  determine  factors  that 
differentiate  their  behavior  with  respect  to  one  another.  Consequently,  the  resultant 
relationship  would  have  to  be  validated  with  a  number  of  different  groups. 


Recommendations  for  Future  Research 

There  are  two  main  areas  for  future  research.  The  first  area  concerns  the  elements 
of  AAR  effectiveness  and  completion  of  AAR  study.  The  second  concerns  the 
requirements  to  implement  the  Inquiry  approach  in  an  AAR  system. 
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The  first  priority  for  future  research  is  to  prove  that  discussion  participation, 
discussion  focus,  learning  reinforcement,  and  timeliness  each  affect  learning  positively. 
This  step  would  validate  the  literature  research  that  developed  the  elements  of 
effectiveness.  This  effort  will  impact  any  future  study  of  AAR  effectiveness  by  clearly 
establishing  a  relationship  between  specific  factors  and  successful  learning  in  the  AAR 
environment. 

A  follow  on  research  effort  would  be  to  increase  the  sample  size  of  AARs  in  the 
study  conducted  in  Chapter  III.  By  redefining  the  variables  lAW  the  findings  in  the 
above  proof  of  principle,  a  clear  articulation  of  what  is  actually  occurring  in  an  AAR  can 
be  achieved.  A  sample  size  of  at  least  50  is  probably  needed  to  draw  conclusions  for  the 
CTC  AARs.  This  would  be  relatively  simple  to  conduct  since  a  portion  of  AARs  are 
videotaped  each  rotation. 

However,  an  idea  of  how  AARs  are  conducted  during  home-station  training  is 
also  needed.  This  will  indicate  how  the  doctrine  and  example  set  by  the  CTCs  are 
implemented.  Sampling  these  AARs  presents  a  couple  of  problems:  1)  few  of  these 
AARs  are  videotaped  and  therefore,  2)  data  collectors  would  have  to  be  present  to 
observe  or  record  the  AAR.  The  presence  of  the  data  collectors  alone  may  affect  the  way 
the  AAR  is  normally  conducted  or  how  die  trainees  normally  act.  An  option  may  be  to 
conduct  a  written  survey  of  unit  leaders  to  determine  what  elements  of  an  effective  AAR 
they  implemented.  To  reduce  the  survey  participant’s  subjective  interpretation  of  the 
questions,  a  videotaped  example  of  each  question  may  need  to  be  provided. 
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The  second  area  of  future  research  is  required  to  support  the  implementation  of 
the  Inquiry  approach.  First  of  all,  the  examples  demonstrating  learning  efficiency  of  the 
Inquiry  approach  need  further  analysis.  Both  Smithtown  and  Sherlock  exhibited  some  of 
the  Inquiry  theory  strategies,  but  not  all  of  them.  Littman  et  al.  (1990)  identified  which 
strategies  were  most  often  used  to  diagnose  student  bugs.  They  did  not  compare  the 
diagnosing  effectiveness  of  each.  Therefore,  the  questions  that  must  be  answered  are:  1) 
Which  strategies  are  most  important?,  2)  Are  all  10  strategies  needed?,  and  3)  Are  the 
goals,  ^ubgoals,  and  control  strategies  still  valid  if  only  some  of  the  questioning  strategies 
are  used? 

The  next  step  to  the  implementation  is  to  prove  that  the  Inquiry  approach  can  be 
generalized  to  affect  collective  performance.^^  This  would  entail  an  experiment  with 
three  groups  of  units.  Each  experimental  group  should  consist  of  at  least  8  “subject” 
units  to  allow  for  a  sufficiently  large  sample.  A  total  of  24  units  would  be  required.  An 
infantry  or  armor  brigade  tasked  to  support  this  experiment  could  provide  27 
homogeneous  platoons  as  well  as  plan,  prepare,  execute,  and  evaluate  the  training 
exercises. 

Table  5.1,  AAR  Technique  Experiment  outlines  the  training  and  AAR  treatment 
methodology.  The  experimental  treatments  should  include:  1)  a  baseline  assessment 
(pre-test)  to  establish  the  unit’s  proficiency  level;  2)  a  training  exercise  with  common 
tasks,  conditions,  and  standards  followed  by  a  group  specific  AAR  treatment;  and  3)  a 

“Collective”  performance  is  military  unit  performance. 
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post-test  assessment  exercise.  Most  likely,  the  pre-test  will  have  to  consist  of  a 
performance  oriented  test  that  is  also  externally  evaluated.'*^ 


Table  5.1,  AAR  Technique  Experiment 


Group  A 

Group  B 

Group  C 

Baseline  assessment 
(Pre-test) 

X 

X 

X 

Standard  training  exercise 

X 

X 

X 

Inquiry  approach  AAR 

X 

CTC  AAR 

X 

External  evaluation  training 
exercise  (Post  test) 

X 

X 

X 

From  this  experiment,  one  can  determine  the  learning  gain  that  results  from  the 
training  and  the  AAR  technique.  The  hypotheses  to  be  tested  are: 

H,:  Employing  the  CTC  AAR  technique  improved  performance  more  than 
training  with  no  AAR. 

Hj:  Employing  the  Inquiry  approach  AAR  improved  performance  more  than  the 
CTC  AAR  technique. 

“External”  means  not  organic  to  the  unit  being  tested.  These  evaluators  should  come  from  another  imit 
and  not  have  an  undefined  interest  in  the  test  results. 
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As  long  as  the  training  conditions  for  each  unit  are  the  same,  the  performance 
improvement  can  be  attributed  to  the  AAR. 

The  last  implementation  step  would  then  be  to  determine  the  best  method  of 
presenting  the  Inquiry  approach  during  the  AAR.  In  Appendix  E,  Proposed  Design,  I 
suggest  an  Inquiry  Tutor  component  to  support  simulation  training  system  AARs. 

Simply,  this  is  an  expert  system  that  outputs  Inquiry  strategies  in  response  to  data  input 
during  the  AAR.  This  requires  an  Inquiry  Tutor  operator,  separate  from  the  AAR  leader, 
unless  an  input  method  that  does  not  detract  from  the  AAR  leader’s  participation 
monitoring  responsibility  can  be  found.  Once  this  form  is  identified,  separate  training 
systems  can  be  designed  to  train  the  Inquiry  approach  with  respect  to  the  AAR.  These 
training  systems  could  be  individual  intelligent  tutoring  systems  or  even  virtual  world 
training  systems  for  collective  AAR  leader  training. 

Epilogue 

Training 

To  implement  the  Inquiry  approach  for  performance  feedback,  there  is  a 
requirement  to  train  unit  leaders.  This  training  should  be  provided  at  home-station  as 
well  as  basic  and  advanced  schools.  The  goal  should  be  to  train  unit  leaders  to  utilize  the 
approach.  This  would  allow  leaders  to  employ  the  approach  in  live  training  AARs  that 
are  vmsupported  with  simulation-based  data  collection  subsystems.  Additionally,  the 
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Inquiry  approach  teaches  individuals  and  organizations  the  meta-cognitive  skills 
necessary  to  learn  efficiently.  This  instruction  will  transfer  to  many  academic  disciplines 
as  Collins  had  found.  Therefore,  soldiers  will  continue  to  realize  the  benefits  well  after 
their  military  careers. 

If  the  study  of  1 7  AAR  tapes  is  indicative  of  the  majority  of  AARs  conducted  in 
the  Army,  then  some  training  must  be  focused  on  increasing  participation.  Argyris’ 
theory  does  include  some  optimism  for  trainee  participation.  A  surprising  point  that  he 
repeatedly  makes  and  supports  with  evidence  is  that  individuals  are  not  aware  that  they 
are  implementing  Model  I  strategies  to  pursue  the  governing  goals.  However,  individuals 
are  aware  of  others  doing  so.  (Argyris,  1994,  Chapters  1  through  8)  According  the 
ArgjTis,  diis  consistency  is  a  factor  in  the  modification  of  Model  I  behaviors  into  Model 
II  behaviors.  A  solution  Argyris  proposes  is  to  assign  a  trained  person  the  responsibility 
to  intervene  in  group  meetings.  Specifically,  he  should  make  people  aware  of  Model  I 
tendencies  by  exposing  the  defensive  reasoning  they  use  to  rationalize  those  tendencies. 
This  person  becomes  a  facilitator  -  a  “summarizer  of  opinions,”  “clarifier  of  feelings,” 
and  “instigator”  of  opposing  arguments.  (Argyris,  1994, 194  and  221)  Not  only  is  this 
solution  ready  made  for  the  AAR  leader  but  is  in  keeping  with  doctrinal  guidance  to 
facilitate  the  discussion. 

The  main  problem  with  the  above  solution  is  that  it  will  take  nearly  all  of  the 
AAR  leader’s  concentration  to  monitor  trainee  behaviors  and  intervene  with  the 
appropriate  action.  But  if  he  does  not  intervene,  the  problem  of  discussion  participation 
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and  its  underlying  causes  will  not  be  redressed.  I  believe  that  this  problem  can  be 
adequately  addressed  with  the  approach  advocated  in  Chapter  IV  and  the  support  of 
current  technology  in  military  simulations  and  AAR  subsystems. 

A  secondary  problem  with  Argyris’  solution  is  that  it  requires  a  person  trained  in 
Model  I  behavior  diagnosis.  There  is  little  doubt  that  the  facilitator  needs  this  training. 
Without  it,  his  feedback  to  the  group  would  be  ineffective.  The  feedback  he  does  provide 
must  have  the  theoretical  basis  of  Argyris’  Model  I  and  II  theory.  The  training 
requirement  may  be  satisfied  by  simply  reading  and  conducting  group  discussions  of  On 
OrpamVatinnal  T, earning  (1994)  with  respect  to  specific  learning  objectives.  This  training 
solution  is  ideal  as  a  block  of  instruction  for  officers  and  NCOs  in  the  company  grade 
branch  courses  (officer  and  NCO  basic  and  advanced  courses). 


Simulations 

R  &  D  efforts  should  focus  on  tools  that  will  aid  articulation  and  elaboration,  as 
well  as  accommodate  immediate  retraining.  AAR  subsystem  functionality  stops  with  the 
display  of  performance  data.  Their  focus  is  on  efficient  and  complete  data  collection  and 
facilitation  of  data  manipulation  for  later  use  and  presentation.  However,  there  is  little 
effort  to  create  and  design  tools  that  support  deficiency  correction  or  reinforce  learning. 
A  concept  of  such  a  tool  is  presented  in  Appendix  E,  Proposed  Design.  It  entails 
connecting  a  constructive  simulation  system  to  a  3D  display.  The  idea  is  to  create  a 


203 


simulation-based  component  on  which  the  trainees  can  test  solutions  in  real  time.  This 
could  aid  trainee  visualization  and  elaboration. 


Task  Analysis 

The  Inquiry  approach  is  a  knowledge  acquisition  technique.  Essentially,  the 
approach  renders  the  elements  needed  to  represent  the  subject  domain  as  a  GOMS 
model.'^''  Users  of  the  Inquiry  approach  break  performance  tasks  into  fundamental  cause- 
effect  relationships.  These  relationships  can  then  be  used  to  identify  the  knowledge  units, 
skills,  and  abilities  diat  describe  the  performance  of  the  task.  In  this  sense,  die  Inquiry 
approach  is  also  an  alternative  knowledge  elicitation  method  that  may  be  used  toward  a 
detailed  cognitive  task  model  of  military  task  performance. 

The  process  to  elicit  and  identify  a  task’s  basic  elements  (cognitive  cues)  is  very 
expensive  in  the  time  and  effort  of  both  knowledge  engineers  and  subject  matter  experts. 
For  (Army)  combat,  collective  tasks,  the  creation  of  realistic  conditions  is  also  expensive. 
Furthermore,  the  literature  reveals  that  detailed  cognitive  task  models  have  not  been  built 
for  most  military  tasks.  Given  a  continuance  of  the  past  and  current  governmental  R&D 
paradigms,  money  will  not  be  invested  to  build  cognitive  task  models  on  which  to  base 
simulation  training  systems.  Hence,  the  creation  of  detailed  cognitive  task  models  can  be 
approached  from  the  other  end  -  iteratively  building  representations  of  performance  from 

^  For  task-based  knowledge  domains,  the  GOMS  model  is  the  knowledge  representation  method  used 
when  conducting  a  cognitive  task  analysis.  (Williams  &  Kotnour,  1993) 
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whatever  the  simulation  system  can  approximate.  This  can  be  likened  to  building  a 
haystack  by  moving  two  to  three  straws  at  a  time. 

Simulation  training  systems  provide  a  medium  on  which  to  observe  and  record  the 
knowledge  acquisition  process  that  the  Inquiry  approach  affords.  The  most  attractive 
characteristic  of  this  potential  task  analysis  method  is  that  the  training  conditions  can  be 
precisely  recreated  from  the  simulation  system’s  data  logger  and  the  knowledge,  skill, 
and  ability  elements  identified  as  critical  can  be  tested.  Hence,  the  important  battlefield 
cues  and  actions  can  be  differentiated  from  those  that  are  irrelevant.  The  AAR  is 
important  because  the  differences  between  “real  life”  and  the  simulation  system  will  be 
highlighted.  The  analysis  should  be  of  the  task  performance  in  replicated  combat 
conditions  or  “acceptable”  approximations  of  those  conditions.  The  universally  accepted 
assumption  is  that  to  be  valid  in  combat,  the  task  performance  must  be  verified  by 
doctrine  and  reality.  Therefore,  any  input  toward  a  cognitive  task  model  can  not  be 
generalized  beyond  the  conditions  of  realistic  simulation. 

Arguably,  the  most  realistic  combat  environments  are  simulated  at  the  Live  Fire 
departments  of  the  CTCs.  As  such,  these  environments  provide  verification  testing  for 
any  simulation-based  training  program  (of  a  limited  number  of  tasks  below  battalion 
level).  The  requirement  for  behavioral  data,  gathered  from  a  simulation  training  system, 
to  be  verified  in  live  training  conditions  cannot  be  emphasized  strongly  enough;  it  is 
criminal  neglect  to  do  otherwise.  Until  simulation  training  systems  are  built  with  respect 
to  validated  and  verified  (detailed)  cognitive  task  models,  a  dire  risk  exists  in  which  tasks 
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are  trained  in  response  to  inappropriate  or  incomplete  conditions.  The  performance  will 
not  transfer  to  combat.  Consequently,  a  high  initial  casualty  rate  will  accompany  a  steep 
re-leaming  curve  as  individuals  and  units  strive  to  adapt  their  behavior  to  survive. 
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APPENDIX  A 


THEORETICAL  FOUNDATIONS 


"Humans  leam  continuously  from  all  tasks  lliey  perform,  at  whatever 
scale  the  task  occurs." 

—  Allen  Newell 

Unified  Theories  of  Cognition,  1992 

"Cognitive  skills  are  realized  by  production  rules." 

-  John  R.  Anderson 

Rules  of  the  Mind,  1993 

Problem  Spaces 

Beginning  in  the  late  1950s,  Herbert  Simon  and  Allen  Newell  conducted  the 
research  and  analysis  that  has  evolved  into  the  most  popular  framework  of  human 
problem  solving  and,  consequently,  the  basis  for  a  number  of  theories  and  models  of 
cognition.  (Simon,  1996)  This  framework  and  the  seminal  theory  of  meta-cognition  are 
presented  in  Human  Problem  Solving  (1972).  They  present  a  theory  of  human  problem 
solving  in  Chapter  14.  In  the  elaboration  of  this  theory,  Simon  and  Newell  describe  the 
features  that  are  common  to  problem  spaces.  These  characteristics  describe  die 
constraints  and  limitations  followed  by  Langley  et  al.  (1987)  in  their  models  of  scientific 
rediscovery.  The  common  characteristics  of  the  problem  space  are  listed  in  Table  A-1, 
Invariant  Features  of  Problem  Spaces. 
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Table  A-1,  Invariant  Features  of  Problem  Spaces 


1 .  Problem  spaces  can  be  represented  as  a  closed  space  of  knowledge  —  a  set  of 
knowledge  states  generated  by  a  finite  set  of  objects,  properties,  and  environmental 
forces  and  their  interrelationships. 

2.  The  knowledge  state  is  typically  moderate  in  size.  It  ranges  from  a  few  dozen 
symbols  to  a  few  hundred. 

3 .  The  set  of  operators  that  can  be  applied  to  the  problem  space  is  small  and  finite 
(finitely  generated). 

4.  The  problem  solver  maintains  one  or  two  alternative  nodes  to  which  he  may  return 
during  solution. 

5.  The  problem  solver  spends  seconds  in  each  selected  knowledge  state  before 
generating  the  next  state. 

6.  The  problem  solver  spends  1  Os  of  minutes  in  a  given  problem  space. 

7.  Problem  solving  takes  place  by  search  in  the  problem  space  -  incrementally 
moving  from  one  knowledge  state  to  another  until  the  desired  knowledge  stated  is 
reached. 

8.  The  search  retraces  (backs-up)  its  solution  path  as  needed. _ 

(Simon  &  Newell,  1972, 810  through  812) 


Learning  and  ACT-R  Theory 

In  Intelligent  Simulation  Training  Systems  Design.  Kent  E.  Williams  (1996) 
reviews  current  cognitive  theories  of  knowledge  representation,  knowledge  acquisition, 
and  problem  solving.  Theories  of  cognition  attempt  to  explain  how  humans  process  and 
recall  information  to  acquire  skill.  Problem  solving,  referred  to  as  meta-cognitive  skill,  is 
how  humans  use  the  information  processed  or  recalled.  (Williams,  1996a)  These 
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cognitive  theories  describe  the  underlying  mechanism  of  the  meta-cognitive  processes 
such  as  Inquiry  theory. 

The  most  successful  model  at  describing  knowledge  acquisition,  performance,  and 
transfer  is  Anderson’s  ACT-R  theory  of  cognition.  (Williams,  1996a)  In  Rules  of  the 
IvrinH  (1993),  Anderson  convincingly  posits  that  the  success  of  modeling  human  learning 
and  performance  with  ACT-R  at  least  validates  its  production  system  structure.  With  this 
conclusion,  ACT-R  is  offered  as  proof  that  human  cognitive  skills  are  realized  hy 
production  rules.  (Anderson,  1993) 

Production  rules  (or  productions)  are  condition-action  pairs  or  if-then  rules. 
(Anderson,  1993, 4  and  5)  The  condition  side  specifies  the  circumstances  under  which 
the  action  side  will  apply.  The  action  side  specifies  what  to  do  if  the  circumstances  are 
satisfied.  These  productions  form  the  basic  units  of  skill.  In  other  words,  productions  are 
what  a  student  learns.  Furthermore,  the  analysis  to  identify  the  production  rules  in  a 
knowledge  domain  and  then  the  teaching  of  those  rules  leads  to  efficient  learning. 
(Anderson,  1993,  Williams,  1996a,  and  Williams,  1995)  The  identification  of  critical 
knowledge  imits  is  the  purpose  of  knowledge  engineering. 

Production  rules  have  four  characteristic  features  with  respect  to  optimizing  their 
utility  in  a  system  of  productions.  These  properties  are  modularity,  abstraction,  goal 
structure,  and  asymmetry.  These  features  are  described  below. 

1 .  Modularity  -  productions  are  independent  of  each  other  and  can  be  added  or 
deleted  as  such.  This  allows  productions  to  combine  in  different  ways  and  produce 
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variations  of  behavior.  Consequently,  solving  a  problem  is  dependent  upon  the 
interactions  among  productions.  Additionally,  the  weakest  production  determines  overall 
performance.  Thus,  sets  of  productions  can  model  erroneous  behavior  by  the  addition  or 
deletion  of  productions.  (Anderson,  1993, 31  through  32) 

The  modularity  feature  also  justifies  the  position  that  cognitive  skills  are  realized 
by  productions.  Productions  are  the  basic  units  of  a  skill  (procedural  knowledge). 
Anderson  proved  this  by  conducting  a  detailed  task  analysis  of  programming  in  LISP.  A 
task  analysis  specifies  the  structure  of  problems  in  a  specific  knowledge  domain.  By 
using  productions  to  describe  each  problem,  the  analysis  identifies  the  units  in  which  skill 
is  acquired.  Anderson  proved  that  LISP  programming  skill  is  acquired  in  production¬ 
sized  units  by  accurately  predicting  how  long  students  would  take  to  learn  to  program  a 
string  of  code.  Programming  skill  increased  by  acquiring  new  productions  and 
strengthening  existing  productions.  (Anderson,  1993,  32  through  34) 

2.  Abstraction  -  productions  can  be  generalized  to  different  situations  because 
they  do  not  require  a  specific  condition  to  fire.  In  other  words,  the  “rules  apply  in  any 
stimulus  condition  that  satisfies  the  pattern  specification  of  the  condition.”  In  ACT-R, 
the  pattern  specification  of  the  condition  is  a  concatenation  of  knowledge  chunks.  The 
configuration  of  these  chunks  is  what  matches  the  condition  side  of  the  production  and 
causes  the  action  side  to  fire.  The  content  of  the  chunks  is  not  a  factor  in  the  pattern 
matching.  (Anderson,  1993, 35  through  37)  Hence,  different  productions  can  share  the 
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same  stimulus  conditions,  and  productions  are  strung  together  to  achieve  complex 
cognition. 

3.  Goal  structured  -  production  rule  conditions  specify  definitive  goal  conditions. 
Depending  on  the  internal  goal  specified  in  each  of  the  conditions,  different  productions 
can  fire  given  die  same  external  situations.  This  allows  the  subject  to  respond  differently 
to  the  same  situation.  (Anderson,  1993,  35  through  37) 

Combining  the  features  of  goal  structure  and  abstraction  allows  Anderson  to 
accurately  predict  skill  transfer.  Anderson  showed  this  transfer  by  measuring  variable- 
coding  errors  in  LISP.  (Anderson,  1993, 36)  In  the  first  lesson,  students  learned  the 
productions  required  to  code  variable  names.  Over  the  next  three  lessons,  students  were 
required  to  generalize  the  productions  to  new  conditions,  but  with  the  same  goal.  Even 
though  the  stimulus  conditions  were  different  in  all  four  lessons,  students  steadily 
improved  performance.  By  lesson  four,  they  averaged  zero  variable-coding  errors. 
However,  in  lesson  five  the  goal  was  changed  and  the  number  of  errors  increased. 
Performance  deteriorated  because  new  productions  had  to  be  learned.  By  counting  the 
number  of  productions  that  are  common  to  two  tasks  and  the  productions  associated  with 
the  goal  changes,  Anderson  is  able  to  accurately  predict  how  long  it  will  take  a  student  to 
generalize  a  skill.  (Anderson,  1993, 36  through  37) 

4.  Asymmetry  -  The  production’s  action  is  determined  from  the  conditions,  but 
the  conditions  carmot  be  determined  from  the  action.  Simply,  the  rule  cannot  reverse 
itself.  The  production  system’s  efficiency  is  derived  from  the  production  rule’s 
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asymmetrical  structure  because  it  reduces  rule  conflict.  Anderson  demonstrated  this  by 
comparing  learning  to  write  LISP  code  and  learning  to  evaluate  it.  Each  group  that  had 
prior  training  in  coding  or  evaluation  performed  the  best  on  those  tasks.  Those  that  had 
prior  training  in  coding  and  the  control  group  (no  training)  performed  equally  well  on  the 
evaluation  tasks.  While  those  that  had  been  trained  in  evaluation  did  better  than  the 
control  group  in  coding  tasks.  (Anderson,  1993,  37  through  38)  The  knowledge  of 
coding  was  a  subset  of  evaluation  and  this  made  the  trained  evaluators  better  at  coding. 
Alternatively,  the  coding  knowledge  could  not  be  reversed  into  the  parent  set  of 
evaluation  knowledge. 

As  a  production  system,  ACT-R  subscribes  to  the  Simon’s  Two  Concept  Theory 
of  Memory  specified  in  the  Human  Information  Processing  Model  of  Cognition. 
(Williams,  1996a,  Anderson,  1993,  and  Simon  &  Newell,  1972)  This  theory  specifies 
that  memory  consists  of  two  long-term  memory  stores,  declarative  storage  and 
procedural  memory.  The  declarative  storage  is  a  knowledge  base  of  facts  while  the 
procedural  memory  stores  productions  -  knowledge  about  how  to  do  something. 
Conscious  thought  is  represented  as  short-term  memory  or  working  memory.  It  receives 
input  and  has  a  capacity  of  five  to  nine  working  memory  elements  or  chunks  of 
information.  A  chunk  is  the  basic  unit  for  the  organization  of  declarative  knowledge. 
(Anderson,  1993,  25  through  29) 
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Working  Memory 


Input  Actions 


Figure  A-1,  Human  Information  Processing  Model 
(Williams,  1996a,  and  Williams,  1995) 


There  are  a  six  properties  that  distinguish  declarative  from  procedural  knowledge. 
The  first  is  reportability.  This  refers  to  the  capability  to  explicitly  declare  or  verbalize  the 
knowledge.  Declarative  knowledge  is  reportable  or  potentially  reportable,  but  procediual 
knowledge  is  not.  (Anderson,  1993, 21)  For  example,  knowing  that  Djakarta  is  in 
Indonesia  is  declarative,  while  the  reading  of  a  map  to  find  Indonesia  is  procedural. 
Humans  have  a  difficult  time  describing  how  they  read. 
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The  second  property  is  associative  priming.  This  is  the  learned  association  of  two 
objects  such  that  given  one,  the  other  is  expected.  For  example,  when  one  hears  the  word 
“computer,”  there  is  priming  for  the  word  “programming.”  Declarative  memory  has  an 
associative  priming  process,  procedural  memory  does  not.  (Anderson,  1993, 21) 

Retrieval  asymmetry  is  the  third  property.  As  stated  before,  this  means  that  given 
the  condition  side  of  a  production,  the  action  side  can  be  retrieved,  but  not  vice  versa.  In 
other  words,  given  the  action  side,  the  conditions  can  not  be  retrieved.  This  property 
applies  only  to  procedural  knowledge.  (Anderson,  1993, 21) 

Acquisition  is  the  fourth  distinguishing  property.  Declarative  knowledge  is 
acquired  directly  from  the  environment.  Procedural  knowledge,  on  the  other  hand,  is 
compiled  from  declarative  knowledge  through  practice.  Procedural  knowledge  cannot  be 
acquired  directly  from  the  environment.  (Anderson,  1993, 22) 

The  fifth  property  is  retention.  For  each  type  of  knowledge,  the  retention 
fimctions  are  independent  of  each  other.  This  property  explains  why  people  can  improve 
the  recall  of  procedural  knowledge  and  at  the  same  time  regress  in  the  recall  of 
declarative  knowledge.  For  example,  the  layout  of  the  typewriter  keyboard  is 
declaratively  memorized  when  first  learning  to  type.  Procedurally,  the  keyboard  layout  is 
known  as  part  of  the  typing  skill.  Thus,  the  knowledge  of  the  keyboard  layout  is 
represented  both  declaratively  and  procedurally.  Furthermore,  as  the  typing  skill 
increases  in  efficiency,  most  people  lose  the  ability  to  recall  the  declarative  representation 
of  the  keyboard  layout.  (Anderson,  1993,  18  and  22) 
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The  final  property  distinguishing  declarative  and  procedural  knowledge  is 
dissociation.  Cognitive  research  has  shown  that  declarative  and  procedural  learning  are 
imcorrelated.  (Anderson,  1993, 22)  However,  Anderson  has  proven  that  procedural 
learning  is  still  dependent  upon  active  declarative  representations.  In  other  words, 
subjects  must  be  able  to  perceive  declaratively  in  order  to  learn  procedurally.  After  the 
productions  are  formed,  ftie  declarative  representation  does  not  have  to  be  retained. 
(Anderson,  1993, 22  through  24)  Hence,  procedural  knowledge  and  declarative 
knowledge  are  dissociated  with  respect  to  retention  in  memory. 

Using  a  computer  simulation  of  ACT-R  and  rational  analysis,  Anderson 
empirically  proved  that  declarative  and  procedural  knowledge  units  are  converted  into 
action  as  a  function  of  their  strengths,  base-level  activations,  probabilities,  and  costs. 
(Anderson,  1993,  69)  More  specifically,  he  was  able  to  determine  that  die  selection  of  a 
production  rule  is  determined  by  the  interactions  of  nine  factors  and  that  this  interaction 
can  be  accurately  modeled  using  Bayseian  statistical  inference.  (Anderson,  1993, 69) 
These  production  rule  factors  are  listed  below.  The  selection  of  a  production  rule  (from 
procedural  storage)  is  dependent  upon: 

•  The  goal  that  is  currently  active;  this  will  spread  activation  to  other  goals 

and  prime  rules. 

•  The  past  history  of  use  of  various  declarative  chunks  -  the  past  probability 

that  the  chunk  might  be  active. 
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•  The  elements  in  the  current  context  -  the  patterns  of  chunks  in  working 

memory  that  activate  and  prime  rules. 

•  The  complexity  of  the  rule.  This  is  the  number  of  chunks  that  must  be 

matched. 

•  The  past  frequency  of  me  of  the  production  rule. 

•  The  past  history  of  success  of  the  production  rule;  with  respect  to  the  goal. 

•  The  amoxmt  of  effort  put  into  solving  the  problem  so  far  -  the  cost  (in  time) 

of  executing  a  production  and  its  consequent  productions. 

•  The  similarity  between  the  goal  state  and  the  state  that  results  from 

applying  the  production  rule.  In  a  hierarchy  of  goals,  this  is  the  distance 
between  the  current  goal  and  the  desired  goal  end-state. 

•  What  other  options  for  behavior  are  available  given  that  the  goal  state  fails. 

(Anderson,  1993, 63) 

These  factors  combine  to  decide  the  base-level  activation  of  a  rule  and  its  strength  of 
association.  The  resultant  activation  is  the  learning  of  that  rule.  As  a  consequence  of 
activation,  this  rule  activates  other  productions  that  also  strengthen  associations. 
(Anderson,  1993, 69  dirough  78) 

The  association  strength  connecting  productions  or  chunks  determines  the  pattern 
that  is  recognized  by  the  condition  side  of  the  declarative  and  procedural  memory. 
(Anderson,  1993, 76)  This  reasoning  leads  to  an  important  effect  that  concerns  learning  - 
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ih&fan  effect.  The  fan  effect  states  that  the  memory  access  spread  from  a  cue  to  a 
knowledge  unit  (declarative  or  procedural)  decreases  as  the  number  of  memories  (links) 
associated  with  that  cue  increases.  (Anderson,  1993, 77)  Hence,  the  exercise  that 
associates  the  cue  with  the  declarative  chunks  in  the  most  ways  will  increase  the  speed  at 
which  the  desired  procedural  or  declarative  knowledge  units  are  recalled.  The  key  is  to 
elicit  the  correct  productions  or  declarative  information  and  reinforce  the  association 
strength.  The  increase  in  associative  strength  then  causes  a  corresponding  increase  in 
activation  level.  This  is  why  skill  practice  and  elaboration  of  context  specific  experience 
improve  performance  of  that  skill. 

More  specifically,  ACT-R  is  a  computational  model  of  productions  with  the 
premise  that  learning  is  based  upon  what  is  already  known  -  prior  experience.  This 
experience  is  stored  in  procedural  memory  and  consists  of  the  production  rules  with 
similar  goals  and/or  conditions.  (Anderson,  1993)  With  more  elaboration  and  practice, 
these  productions  become  situation  specific,  allowing  the  experience  to  be  situation- 
based.  The  subject  learns  by  iteratively  applying  that  experience  to  a  given  problem. 

ACT-R’ s  implications  for  skill  acquisition  are  summarized  below.  These  derive 
directly  from  the  argument  that  human  cognitive  skills  are  realized  by  production  rules. 

1.  The  knowledge  underlying  a  skill  begins  in  an  initial  declarative  form 
(an  elaborated  example),  which  must  be  interpreted  (problem 
solving  by  analogy)  to  produce  performance. 
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2.  As  a  fimction  of  its  interpretive  execution,  this  skill  becomes 

compiled  into  production-rule  form. 

3.  With  practice,  individual  production  rules  acquire  strength  and 

become  more  attuned  to  the  circumstances  in  which  they  apply; 

they  become  situation  based. 

4.  Learning  complex  skills  can  be  decomposed  into  the  learning 

fimctions  associated  with  individual  production  rules. 

(Anderson  et  al.,  1993, 143) 

Given  that  production  units  are  the  basic  vmits  of  skill,  performing  a  detailed  task 
analysis  with  respect  to  productions  can  improve  performance.  More  efficient  learning  is 
achieved  by  using  this  analysis  (of  the  production  units)  to  specifically  incorporate  the 
identified  knowledge  chunks  into  the  curriculum.  (Anderson,  1993,  and  Williams,  1996a) 
Thus,  the  question  arises  of  why  task  analyses,  in  production  level  detail,  have  not  been 
employed  in  training  systems?  The  short  answer  to  this  question  is  that  the  task  analysis 
process  is  an  extremely  expensive  venture  in  both  time  and  resources.  (Williams,  1996a, 
Williams,  1995,  and  Chapman  &  Allen,  1994)  Despite  the  expense,  computer  tools  that 
aid  and  facilitate  the  process  of  expert  knowledge  elicitation  and  representation  have  been 
and  are  being  developed.  (Chapman  &  Allen,  1994,  and  Williams,  1996b)  In  the 
meantime,  less  expensive  methods  must  be  used  to  learn  more  efficiently.  One  such 
method  is  Inquiry  theory. 
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APPENDIX  B 


REGRESSION  ANALYSIS 


"We  reason  with  scant  evidence,  vague  concepts,  heuristic  syllogisms, 
tentative  facts,  rules  of  thumb,  principles  shot  through  with  exceptions, 
and  an  inarticulable  pantiieon  of  inexact  intuitions,  hxmches,  suspicions, 
beliefs,  estimates,  guesses  and  the  like." 

-Kosko,1992 


Univariate  multiple  regression  by  backwards  elimination  was  used  to  statistically 
confirm  any  trends  found  in  the  data.  This  analysis  was  conducted  with  MINITAB  for 
Windows  software.  All  regression  models  were  tested  to  a  statistical  significance  level 
(a)  of  0.05.  Multiple  regression  analysis  was  conducted  for  each  of  the  quantitative 
measures  for  which  data  was  recorded.  This  analysis  indicated  which  factor(s)  seemed  to 
have  a  large  or  small  effect  on  a  selected  dependent  variable.  Stepwise  (forward) 
regression  was  used  to  verify  the  significant  terms.  Finally,  Eigen  analysis  was  used  to 
test  for  multicollinearity. 

The  regression  model  outputs  are  labeled  as  output  tables  and  include:  the 
regression  model;  a  list  of  independent  variables  and  their  corresponding  coefficients  (P), 
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standard  errors,  test  statistics  for  testing  Pj  =  0,  and  observed  significance  p-values;  the 
model  adequacy  measurements  and  R^-adjusted;  an  analysis  of  variance  table;  and  the 
eigenvalues  for  the  X’X  matrix.  The  variance  inflation  factor  (VIF)  is  computed  for  the 
small  eigenvalues.'*’  A  VIF  >100  was  considered  a  large  value  and  resulted  in  the 
regressor  variable(s)  being  removed  fi:om  the  model.  Each  output  table  (B-2  through  B- 
10)  corresponds  to  the  summaries  in  Tables  3.11,  Discussion  Participation  Factors,  and 
3.12,  Discussion  Focus  Factors. 

A  list  of  dependent  and  independent  variables  and  their  definitions  is  given  in 
Table  B-1,  Factor  Definitions  (next  page).  The  raw  data  for  these  variables  are  in  Table 
3.5,  Discussion  Participation  Data,  Table  3.6,  Discussion  Focus  Data,  Table  3.7,  AAR 
Leader  Actions  Data,  and  Table  3.8,  Unit  and  Other  Actions  Data. 


The  VIF  for  the  smallest  eigenvalue  is  the  ratio  of  the  largest  eigenvalue  to  the  smallest  eigenvalue.  The 
VIF  for  the  next  smallest  value  is  simply  the  ratio  of  the  largest  eigenvalue  to  the  next  smallest  eigenvalue. 
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Table  B-1,  Factor  Definitions 


Factor 

Definition 

Type 

PARI 

Unit :  AAR  leader  utterance  ratio 

Dependent 

PAR2 

%  of  unit  members  participating  in  the  discussion 

Dependent 

PARS 

Ratio  of  Unit  comments  :  AAR  leader  questions 

Dependent 

PAR4 

Discussion  amongst  the  unit  during  the  AAR 

Dependent 

PARS 

#  Questions  asked  by  the  unit  during  the  discussion 

Dependent 

TIMEl 

%  of  total  AAR  time  used  for  discussion 

Dependent 

DuP 

#  solutions  planned  specified  by  the  unit 

Dependent 

DuD 

#  solutions  developed  specified  by  the  unit 

Dependent 

Dtp 

Total  #  solutions  planned  specified  (by  unit  &  AAR  leader) 

Dependent 

DtD 

Total  #  solutions  developed  specified  (by  unit  &  AAR  leader) 

Dependent 

A 

AAR  leader  used  focused,  open  questions 

Independent 

B 

AAR  leader  used  focused,  leading  questions 

Independent 

C 

AAR  leader  provided  participation  feedback  during  the  discussion 

Independent 

D 

AAR  exceeded  specified  time  limit 

Independent 

E 

AAR  leader  established  a  participatory  climate  in  the  introduction 

Independent 

F 

AAR  leader  used  specific  BDA  during  the  discussion 

Independent 

G 

AAR  leader  used  sketches  and  maps  during  the  discussion 

Independent 

H 

#  questions  asked  by  the  AAR  leader 

Independent 

I 

#  questions  asked  by  unit 

Independent 

J 

Unit  had  a  good  attitude 

Independent 

K 

Unit  had  a  poor  attitude 

Independent 

L 

Unit  leader  aided/facilitated  discussion 

Independent 

M 

Unit  leader  hindered  discussion 

Independent 

N 

Unit  specified  solutions  planned 

Independent 

O 

Unit  specified  solutions  developed 

Independent 
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Output  B-2,  Unit :  AAR  Leader  Utterance  Ratio  (PARI) 


PARI  =  0.0892  +  0.969E  -  0.541H  +  0.820AB  -  0.4470  +  0.517CG  -  0.269BE 


Predictor 

Coef  Stdev 

t-ratio 

P 

Constant 

0.08919  0.08328 

1.07 

0.309 

E 

0.9686  0.1269 

7.64 

0.000 

H 

-0.54058  0.07860 

-6.88 

0.000 

AB 

0.8196  0.1325 

6.19 

0.000 

0 

-0.4465  0.1182 

-3.78 

0.004 

CG 

0.5166  0.1869 

2.76 

0.020 

BE 

-0.2692  0.1102 

CM 

1 

0.035 

s  =  0.1506 

R-sq  =  94.0% 

R-sq(adj)  = 

=  90.3% 

Analysis  of 

Variance 

SOURCE 

DF  SS 

MS 

F 

Regression 

6  3.52533 

0.58755 

25.90 

Error 

10  0.22687 

0.02269 

Total 

16  3.75220 

SOURCE 

DF  SEQ  SS 

E 

1  0.47641 

H 

1  1.47845 

AB 

1  0.88105 

0 

1  0.42516 

CG 

1  0.12900 

BE 

1  0.13525 

Unusual  Observations 

Obs .  E 

PARI  Fit 

Stdev. Fit 

Residual 

14  1.00 

2.0100  2.0100 

0.1506 

0.0000 

P 

0.000 


St .Resid 
*  X 


X  denotes  an  obs.  whose  X  value  gives  it  large  influence. 


Eigenvalues  and  variance  inflation  factors  for  PARI: 

2.25269  1.20224  0.48730  0.47225  0.20091  0.11025  0.02740 

VIF  =  20  VIF  =  36 
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Output  B-3,  %  Unit  Participation  (PAR2) 


PAR2  =  83.2  - 12.8  D  +  18.9  G  +  16.8  AB  -  18.7  BG 


Predictor 

Coef 

Stdev 

t-ratio 

P 

Constant 

83.205 

4.247 

19.59 

0.000 

D 

-12.769 

5.353 

-2.39 

0.034 

G 

18.897 

6.515 

2.90 

0.013 

AB 

16.795 

7.709 

2.18 

0.050 

BG 

-18.72 

10.35 

-1.81 

0.096 

s  -  9.098 

R-sq 

=  64 

.5% 

R-sq(adj)  = 

52.7% 

Analysis  of 

Variance 

SOURCE 

DF 

SS 

MS 

F 

Regression 

4 

1804. 

82 

451.21 

5.45 

Error 

12 

993. 

18 

82.76 

Total 

16 

2798. 

00 

SOURCE 

DF 

SEQ 

SS 

D 

1 

491. 

79 

G 

1 

903. 

41 

AB 

1 

139. 

05 

BG 

1 

270. 

58 

Unusual  Observations 

Obs.  D  PAR2  Fit  Stdev.Fit  Residual 

10  0.00  100.00  83.21  4.25  16.79 


R  denotes  an  obs.  with  a  large  st.  resid. 


Eigenvalues  and  VIF  for  PAR2 

1.87160  0.75107  0.26257  0.17057  0.03394 

VIF  =  55 


P 

.010 


St .Resid 
2.09R 
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Output  B-4,  Unit  Comment :  AAR  Leader  Question  Ratio  (PAR3) 


PARS  =  0.848  +  2.00CF  +  0.827A  -  0.699H  +  0.617E  -  0.634N  +  0.571BG 


Predictor 

Coef 

Stdev 

t-ratio 

P 

Constant 

0.8476 

0.1127 

7.52 

0.000 

CF 

2.0023 

0.2966 

6.75 

0.000 

A 

0.8267 

0.1724 

4.80 

0.000 

H 

-0.6986 

0.1118 

-6.25 

0.000 

E 

0.6173 

0.1359 

4.54 

0.000 

N 

-0.6339 

0.2401 

-2.64 

0.025 

BG 

0.5715 

0.2061 

2.77 

0.020 

s  =  0.2174 

R-sq 

=  97.1% 

R-sq(adj)  = 

^  95.4% 

Analysis  of 

Variance 

SOURCE 

DF 

SS 

MS 

F 

Regression 

6 

15.9576 

2.6596 

56.26 

Error 

10 

0.4728 

0.0473 

Total 

16 

16.4304 

SOURCE 

DF 

SEQ  SS 

CF 

1 

8.6230 

A 

1 

3.6959 

H 

1 

2.3117 

E 

1 

0.7730 

N 

1 

0.1905 

BG 

1 

0.3634 

Unusual  Observations 

Obs.  CF  PAR3  Fit  Stdev.Fit  Residual  St.Resid 

14  1.00  4.6600  4.6600  0.2174  0.0000  *  X 

X  denotes  an  obs.  whose  X  value  gives  it  large  influence. 


Eigenvalues  and  VIF  for  PAR3 

2.40581  1.37293  0.80669  0.61600  0.18658  0.10976  0.03277 

VIF  =  73 


Output  B-5,  Number  Questions  Asked  by  the  Unit  (PARS) 


PARS  =  -  0.938  +  1.16  C  +  0.493  L  +  0.474  AE 


Predictor 

Coef 

Stdev 

t-ratio 

P 

Constant 

-0.93757 

0.03297 

-28.43 

0.000 

C 

1.1594 

0.1189 

9.75 

0.000 

L 

0.4928 

0.1189 

4.14 

0.001 

AE 

0.4738 

0.1081 

4.38 

0.001 

s  =  0.1189 

R-sq 

==  97.4% 

R-sq(adj)  • 

=  96.7% 

Analysis  of 

Variance 

SOURCE 

DF 

SS 

MS 

F 

P 

Regression 

3 

6.7556 

2.2519 

159.33 

0.000 

Error 

13 

0.1837 

0.0141 

Total 

16 

6.9393 

SOURCE 

DF 

SEQ  SS 

C 

1 

5.6138 

L 

1 

0.8703 

AE 

1 

0.2715 

Unusual  Observations 

Obs .  C 

I 

Fit  Stdev. Fit 

Residual 

St .Resid 

3  1.00 

0.8841 

0.6957 

0.1030 

0.1884 

3.17RX 

12  0.00 

0.2174 

0.0290 

0.1030 

0,1884 

3.17RX 

13  0.00 

-0.6522 

-0.4638 

0.1030 

-0.1884 

-3.17RX 

14  1.00 

1.0000 

1.1884 

0.1030 

-0.1884 

-3,17RX 

R  denotes  an  obs.  with  a  large  st.  resid. 

X  denotes  an  obs,  whose  X  value  gives  it  large  influence. 


Eigenvalues  and  VIF  for  PARS 

1.627S2  1.00000  0.22344  0.0S288 

VIF  =  31 
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Output  B-6,  Discussion  Amongst  the  Unit  (PAR4) 


PAR4  =  0.200  -  0.200  E  +  1.00  N  +  0.800  AF  -  0.800  FG 


Predictor 

Coef 

St  dev 

t-ratio 

P 

Constant 

0.2000 

0.1155 

1.73 

0.109 

E 

-0.2000 

0.1495 

-1.34 

0.206 

N 

1.0000 

0.2080 

4.81 

0.000 

AF 

0.8000 

0.2828 

2.83 

0.015 

FG 

-0.8000 

0.3511 

00 

CM 

CM 

1 

0.042 

s  =  0.2582 

R-sq  = 

77.3% 

R-sq(adj)  = 

69.8% 

Analysis  of  Variance 


SOURCE 

DF 

SS 

MS 

Regression 

4 

2.72941 

0.68235 

Error 

12 

0.80000 

0.06667 

Total 

16 

3.52941 

SOURCE 

DF 

SEQ  SS 

E 

1 

0.01426 

N 

1 

2.18182 

AF 

1 

0.18713 

FG 

1 

0.34620 

F  p 

10.24  0.001 


Unusual  Observations 


Obs . 

E 

PAR4 

Fit 

Stdev.Fit 

Residual 

St . Resid 

5 

0.00 

1.0000 

1.0000 

0.2582 

0.0000 

*  X 

16 

0.00 

1.0000 

0.2000 

0.1155 

0.8000 

3.46R 

R  denotes  an  obs .  with  a  large  st.  resid. 

X  denotes  an  obs.  whose  X  value  gives  it  large  influence. 


Eigenvalues  for  PAR4 

2.88174  0.75468  0.37030  0.19047  0.03524 

VIF  =  96 
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Output  B-7,  Unit  Specified  Solutions  Developed  (DuD) 


DuD  =  0.326  +  2.73  A  +  1.75  D  -  1.53  F  - 1.79  G  +  2.39  L  -  2.06  M 


Predictor 

Coef 

Stdev 

t-ratio 

P 

Constant 

0.3258 

0.1910 

1.71 

0.119 

A 

2.7252 

0.3825 

7.12 

0.000 

D 

1.7488 

0.3588 

4.87 

0.000 

F 

-1.5309 

0.4667 

-3.28 

0.008 

G 

-1.7890 

0.3703 

-4.83 

0.000 

L 

2.3945 

0.4703 

5.09 

0.000 

M 

-2.0618 

0.3521 

-5.86 

0.000 

s  =  0.4323 

R-sq 

=  92.2% 

R-sq(adj)  = 

:  87.5% 

Analysis  of 

Variance 

SOURCE 

DF 

SS 

MS 

F 

Regression 

6 

22.1315 

3.6886 

19.74 

Error 

10 

1.8685 

0.1868 

Total 

16 

24.0000 

SOURCE 

DF 

SEQ  SS 

A 

1 

12.6212 

D 

1 

0.0005 

F 

1 

0.0833 

G 

1 

0.0976 

L 

1 

2.9208 

M 

1 

6.4081 

Unusual  Observations 

Obs.  A  DuD  Fit  Stdev.Fit  Residual 

11  0.00  1.000  0.286  0.264  0.714 


R  denotes  an  obs.  with  a  large  st.  resid. 


Eigenvalues  and  VIF  for  DuD 

2.66848  1.05408  0.84912  0.41545  0.28066  0.11595  0.03039 

VIF  =  88 


P 

.000 


St .Resid 
2.09R 
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Output  B-8,  Total  Solutions  Developed  (DtD) 


DtD=  1.63  +  1.78  C+ 1. 17  F 


Predictor 

Coef 

Stdev 

t-ratio 

P 

Constant 

1.6322 

0.2598 

6.28 

0.000 

C 

1.7816 

0.7082 

2.52 

0.025 

F 

1.1724 

0.5379 

2.18 

0.047 

s  =  0.9160 

R-sq 

=  50.6% 

R-sq{adj)  = 

=  43.5% 

Analysis  of 

Variance 

SOURCE 

DF 

SS 

MS 

F 

P 

Regression 

2 

12.0176 

6.0088 

7.16 

0.007 

Error 

14 

11.7471 

0.8391 

Total 

16 

23.7647 

SOURCE 

DF 

SEQ  SS 

C 

1 

8.0314 

F 

1 

3.9862 

Unusual  Observations 

Obs .  C 

DtD 

Fit 

Stdev. Fit 

Residual 

St .Resid 

3  1.00 

4.000 

3.414 

0.701 

0.586 

0.99  X 

14  1.00 

4.000 

4.586 

0.701 

-0.586 

-0.99  X 

X  denotes  an  obs.  whose  X  value  gives  it  large  influence. 
Regression  Analysis 


Eigenvalues  and  VIF  for  DtD 

0.635384  0.333333  0.054271 

VIF  =  12 
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Output  B-9,  Unit  Specified  Solutions  Planned  (DuP) 


DuP  =  0.481  +  0.5221  +  1.47JL  -  1.41CF  -  0.1 12F  +  0.0479B 


Predictor 

Coef 

Stdev 

t-ratio 

P 

Constant 

0.48148 

0.01835 

26.24 

0.000 

I 

0.52166 

0.01742 

29.94 

0.000 

JL 

1.46950 

0.03937 

37.33 

0.000 

CF 

-1.40826 

0.04356 

-32.33 

0.000 

F 

-0.11231 

0.02244 

-5.01 

0.000 

B 

0.04793 

0.01708 

2.81 

0.017 

s  =  0.02925 

R-sq  = 

99.8% 

R-sq ( ad j)  = 

99.7% 

Analysis  of  Variance 


SOURCE 

DF 

SS 

MS 

F 

P 

Regression 

5 

5.0494 

1.0099 

1180.30 

0.000 

Error 

11 

0.0094 

0.0009 

Total 

16 

5.0588 

SOURCE 

DF 

SEQ  SS 

I 

1 

3.3837 

JL 

1 

0.7238 

CF 

1 

0.9165 

F 

1 

0.0187 

B 

1 

0.0067 

Unusual  Observations 

Obs.  I 

DuP 

Fit 

Stdev. Fit 

Residual 

St .Resid 

3  0.88 

1.00000 

0.99059 

0.02911 

0.00941 

3.32RX 

12  0.22 

2.00000 

2.00000 

0.02925 

0.00000 

*  X 

14  1.00 

1.00000 

1.00000 

0.02925 

0.00000 

*  X 

R  denotes  an  obs.  with  a  large  st.  resid. 

X  denotes  an  obs.  whose  X  value  gives  it  large  influence. 


Eigenvalues  for  DuP 

2.85940  1.62915  0.66770  0.42361  0.09194  0.03445 

VIF  =  83 
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Output  B-10,  Total  Solutions  Planned  (DtP) 


Dtp  =  1.69  +  0.167  B  -  2.43  C  -  0.326  F  +  1.84 1 


Predictor 

Coef 

Stdev 

t-ratio 

P 

Constant 

1.6917 

0.1211 

13.97 

0.000 

B 

0.16715 

0.06162 

2.71 

0.019 

C 

-2.4295 

0.2184 

-11.12 

0.000 

F 

-0.32636 

0.08422 

-3.87 

0.002 

I 

1.8405 

0.1231 

14.96 

0.000 

s  =  0.1060 

R-sq 

=  97.3% 

R-sq ( ad j)  • 

=  96.4% 

Analysis  of 

Variance 

SOURCE 

DF 

ss 

MS 

F 

P 

Regression 

4 

4.9241 

1.2310 

109.65 

0.000 

Error 

12 

0.1347 

0.0112 

Total 

16 

5.0588 

SOURCE 

DF 

SEQ  SS 

B 

1 

1.3445 

C 

1 

0.5143 

F 

1 

0.5538 

I 

1 

2.5114 

Unusual  Observations 

Obs .  B 

Dtp 

Fit  i 

Stdev. Fit 

Residual 

St .Resid 

12  1.00 

2.0000 

1.9326 

0.1042 

0.0674 

3 . 4  6RX 

13  0.00 

0.0000 

0.1651 

0.0700 

-0.1651 

-2.08R 

R  denotes  an  obs.  with  a  large  st.  resid. 


Eigenvalues  for  DtP 

2.85940  1.62915  0.66770  0.42361  0.09194  0.03445 

VIF  =  83 
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Output  B-1 1,  %  Discussion  Time  (TIMEl) 


TIMEl  =  292  -  406C  +  23.9D  -  27.2E  +  27.6H  +  2281  -  10.7K  +  24.40 


-18.1AB- 

218AL-85.1FG 

-446HL 

Predictor 

Coef 

Stdev 

t-ratio 

P 

Constant 

292.35 

72.58 

4.03 

0.010 

c 

-405.9 

141.1 

-2.88 

0.035 

D 

23.922 

8.128 

2.94 

0.032 

E 

-27.16 

11.35 

-2.39 

0.062 

H 

27.580 

7.894 

3.49 

0.017 

I 

228.22 

78.30 

2.91 

0.033 

K 

-10.745 

8.262 

-1.30 

0.250 

0 

24.44 

10.38 

2.36 

0.065 

AB 

-18.13 

14.42 

-1.26 

0.264 

AL 

-218.3 

100.6 

-2.17 

0.082 

FG 

-85.12 

25.72 

-3.31 

0.021 

HL 

-446.0 

187.7 

-2.38 

0.063 

s  =  12. 

01 

R-sq 

=  86.2% 

R-sq(adj)  = 

55.7% 

Analysis  of 

Variance 

SOURCE 

DF 

SS 

MS 

F  P 

Regression 

11 

4494.5 

408.6 

2.83  0.130 

Error 

5 

721.2 

144.2 

Total 

16 

5215.8 

SOURCE 

DF 

SEQ  SS 

C 

1 

34.2 

D 

1 

0.1 

E 

1 

68.5 

H 

1 

1001.7 

I 

1 

621.9 

K 

1 

816.2 

0 

1 

361.1 

AB 

1 

10.6 

AL 

1 

0.0 

FG 

1 

766.0 

HL 

1 

814.2 

*  NOTE 

*  c 

is  highly 

correlated 

with  other 

predictor  variables 

*  NOTE 

*  I 

is  highly 

correlated 

with  other 

predictor  variables 

*  NOTE 

*  AL 

is  highly 

correlated 

with  other 

predictor  variables 

Unusual  Observations 


Obs . 

c 

TIMEl 

Fit 

Stdev. Fit 

Residual  St.Resid 

3 

1.00 

85.00 

85.00 

12.01 

0.00  *  X 

12 

0.00 

90.00 

90.00 

12.01 

0.00  *  X 

13 

0.00 

52.00 

52.00 

12.01 

0.00  *  X 

14 

1.00 

71.00 

71.00 

12.01 

0.00  *  X 

X  denotes  an 

obs .  whose 

X  value 

gives  it 

large  influence. 
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APPENDIX  C 


CURRENT  AAR  SYSTEMS  IN  TRAINING  SIMULATIONS 
Small  Unit  Simulation  Training  Systems 
Cvirrently,  the  Anny’s  small  unit  simulation  training  systems  are  SIMNET  and 
CCTT.  Both  train  platoon  and  company  collective  tasks  in  a  virtual  environment. 
SIMNET  is  the  legacy  system  and  CCTT,  its  replacement,  is  the  first  of  a  number  of 
virtual  training  systems  that  will  form  the  Combined  Arms  Tactical  Trainer  (CATT).  In 
completion,  the  CATT  will  consist  of  the:  CCTT  for  mechanized  infantry,  armor,  and 
cavalry  units;  Aviation  Combined  Arms  Tactical  Trainer  (AVCATT)  for  Army  aviation 
units;  Fire  Support  Combined  Arms  Tactical  Trainer  (FSCATT)  for  field  artillery  and 
mortar  units;  Air  Defense  Combined  Arms  Tactical  Trainer  (ADCATT)  for  air  defense 
units;  and  Engineer  Combined  Arms  Tactical  Trainer  (ENCATT)  for  engineer  units. 
(STRICOM,  1996a)  Each  training  system  will  be  Distributed  Interactive  Simulation 
(DIS)  compliant  and  consist  of  networked,  high  fidelity,  manned  simulators,  Semi- 
Automated  Forces  (SAP),  support  workstations,  computer  networks  and  protocols,  and  an 
AAR  system.  (STRICOM,  1996b) 
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AAR  Systems  in  Training  Simulations 


Larry  Meliza’s  research  report,  Standardizing  Army  After  Action  Review  Systems 
(1996),  is  the  most  complete  and  thorough  source  reviewing  Army  AAR  systems  and 
efforts  for  small  unit  training  systems.  Meliza  conducted  the  research  to  support 
development  of  the  needs  and  capabilities  for  the  Standard  Army  After  Action  Review 
System  (STAARS).  To  complete  this  research,  he  reviewed  the  relevant  literature, 
published  and  impublished  findings  of  three  Army-wide  AAR  conferences,  and  examined 
two  ongoing  AAR  system  development  efforts  in  relation  to  a  legacy  AAR  system.  The 
Army  Training  and  Analysis  Feedback  System  (ATAFS)  and  the  Simulation  Training 
Integrated  Performance  Evaluation  System  (STRIPES)  were  compared  to  the 
development  of  the  Unit  Performance  Assessment  System  (UPAS)  to  clarify  the  needed 
capabilities  of  STAARS.  Many  of  Meliza’s  findings  were  used  by  the  National 
Simulation  Center  (NSC)  as  capability  specifications  and  rationale  for  the  STAARS 
Operational  Requirements  Document  (ORD).  (Meliza,  1996a,  viii) 

The  review  of  Meliza’s  research,  UPAS,  ATAFS,  STRIPES,  the  CCTT  AAR 
system,  and  STAARS,  reveals  that  these  systems  are  primarily  concerned  with  reducing 
the  workload  associated  with  preparing  the  AAR  and  formatting  the  information  products 
that  result  from  the  preparation.  In  other  words,  these  systems  focus  on  automating  data 
collection,  compilation,  and  display  to  reduce  the  amount  of  time  it  takes  a  trainer/AAR 
leader  to  prepare  an  AAR  for  presentation.  In  this  sense,  these  AAR  systems  are 
specifically  designed  to  ensure  the  AAR’s  timeliness  through  preparation  efficiency. 
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The  need  for  preparation  efficiency  is  driven  by  two  reasons.  As  Meli2a  points 
out,  lower  echelon  AARs  precede  higher  echelon  AARs  so  that  the  results  of  the  first  can 
be  used  as  input  for  the  second.  Thus,  to  start  higher  echelon  AARs  in  a  timely  manner, 
lower  level  AARs  must  be  completed  as  soon  as  possible.  The  Mounted  Warfare 
Training  Simulation  Center  at  FT  Knox  tries  to  begin  platoon  AARs  10  minutes  after  the 
end  of  a  training  exercise.  (Meliza,  1996a,  27)  Another  reason  to  rush  preparation  is  that 
units  may  need  to  conduct  subsequent  exercises  on  the  trainer  and  therefore  need  the 
AAR  system  to  collect  data  for  their  exercise.  Currently,  none  of  the  AAR  systems  have 
the  requirement  to  nor  can  they  display  a  presentation  for  a  previous  exercise  while 
collecting  data  from  a  separate  running  exercise. 

It  is  apparent  from  each  AAR  system’s  purpose  statement  and  system  description 
that  none  is  designed  to  directly  improve  effectiveness  with  respect  to  discussion 
participation  and  timely  learning  reinforcement.  (US  ARI,  1997,  BCTD,  1996a, 
LORAL/Army  IDT,  1996,  LB&M,  1996,  LORAL,  1994,  and  Meliza  et  al.,  1992a) 
However,  these  systems  do  impact  AAR  effectiveness  with  respect  to  conducting  a  timely 
AAR  and  discussion  focus.  Well  designed  data  displays  will  aid  units  in  determining 
what  is  critical  and  what  happened.  The  more  quickly  ground  truth  can  be  represented, 
the  faster  the  xmit  can  focus  on  why  it  happened  and  how  to  do  it  better.  This  capability 
greatly  enhances  the  focus  of  discussion  to  solve  performance  problems. 

Except  for  STAARS,  none  of  the  systems  propose  tools  to  aid  in  determining  how 
to  perform  a  task  better.  STAARS  is  requiring  the  capability  to:  “translate  lessons 
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learned  from  the  CALL  [Center  for  Army  Lessons  Learned],  Battle  Command  Battle  Lab 
experiments,  CTC  rotations,  and  other  sources  into  leader  development  and  collective 
training  concepts,  methods  and  strategies  and  revised  doctrine  and/or  tactics,  techniques, 
and  procedures.”  (BCTD,  1996a,  paragraph  3.a.2.a.6)  Although  this  capability  is  not 
identified  for  use  during  the  AAR,  it  is  conceivable  that  if  information  is  collected,  then  it 
could  be  stored  and  updated  for  use  in  the  AAR  also. 

The  retraining  of  specific  tasks  is  also  not  considered  to  be  needed  as  an  AAR 
system  capability.  None  of  the  systems  require  a  capability  to  reinforce  what  was  learned 
during  the  AAR.  This  is  understandable  given  that  the  popular  interpretation  of  the 
training  model  paradigm  concludes  the  cycle  with  the  AAR.  As  stated  before,  this  must 
change  in  order  to  reinforce  the  learning  that  occurs  during  the  AAR. 


STAARS 

The  principal  proponents  for  STAARS  are  the  Deputy  Chief  of  Staff  for  Training 
(DCST)  of  the  Training  and  Doctrine  Command  (TRADOC)  and  the  NSC.  They  are 
charged  to  develop  a  standard  AAR  system  to  “support  training,  mission  rehearsals,  and 
research/experimentation  at  all  echelons  from  individual  through  Echelons  Above  Corp 
(EAC)  with  packages  of  AAR  products  standardized  by  echelon  and  Battlefield 
Operating  System  (BOS).”  (BCTD,  1996a)  The  intent  for  STAARS  is  that  it  satisfy  fiie 
AAR  needs  across  the  Training  Exercises  and  Military  Operations  (TEMO),  Research, 
Development,  and  Acquisition  (RDA)  community,  and  Advanced  Concepts  and 
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Requirements  (ACR)  domains  to  reduce  system  redundancy  and  provide  standard  formats 
for  data  collecting  and  storage  for  training,  testing,  evaluating,  and  analyzing  data  to 
support  the  total  Army.  (BCTD,  1996a) 

The  STAARS  concept  supports  training  as  a  three  tier  system.  The  first  tier 
would  provide  automated,  standard  AAR  products  derived  from  “the  commander’s 
Mission  Essential  Task  List  (METL)  for  training  events.”^*  (BCTD,  1996a)  This 
automates  preparation  for  the  AAR  leader  by  automatically  producing  the  AAR  products 
associated  with  the  tasks  to  be  trained.  The  second  tier  provides  automated  AAR 
products  with  a  selection  menu  for  advanced  or  additional  analysis.  (BCTD,  1996a)  This 
tier  would  be  mostly  used  by  the  R&D  community  for  statistical  analysis  and  verification, 
validation,  and  accreditation.  The  last  tier  provides  the  user  the  ability  to  build 
customized  AAR  products.  (BCTD,  1996a)  The  primary  output  of  the  STAARS  is  the 
standardized  training  exercise  data  that  will  be  stored  in  the  Army  Training  Digital 
Library  (ATDL).  (BCTD,  1996a) 

Currently,  STAARS  exists  conceptually  on  paper  as  a  mission  needs  statement 
and  a  plan  of  action.  For  STAARS,  success  as  a  system  is  defined  as  achieving:  the 
standardization  of  all  data  elements  I  AW  the  DoD  Data  Dictionary;  the  sharing  of  these 
standardized  data  elements  among  the  ACR,  RDA,  and  TEMO  domains;  access  of  data 
across  the  total  Army;  a  reduction  of  training  and  event  evaluation  and  assessment  time; 

BCTD  and  NSC  improperly  use  the  term  METL  here.  A  unit’s  METL  is  an  “unconstrained  [with  respect 
to  resources]  statement  of  tasks  required  to  accomplish  wartime  missions.”  (DA,  FM  25-101, 2-2)  Trying 
to  focus  the  evaluation  and  assessment  on  the  whole  METL  for  each  training  exercise  is  unrealistic  and  is 
an  extremely  inefficient  means  of  improving  performance. 
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and  the  use  of  the  standardized  data  for  analysis  in  support  of  the  ACR’s,  RDA’s,  and 
TEMO’s  purposes.  (BCTD,  1996a)  Taking  this  end  state  by  itself,  one  may  think  that 
STAARS  only  addresses  information  management  problems.  The  only  mention  of  the 
training  management  perspective  is  STAARS’s  aim  to  make  event  evaluation  and 
assessment  time  more  efficient.  The  STAARS  Action  Plan  does  not  address 
effectiveness,  of  anything. 

A  major  problem  that  STAARS  is  seeking  to  remedy  is  the  standardization  of 
AAR  information.  Currently,  AAR  products  are  not  standardized  across  or  within  live, 
virtual,  or  constructive  training  systems.  This  non-standardization  of  products  impact  the 
AAR  and  simulation  integration.  If  multiechelon  training  is  being  conducted  on  live  and 
virtual  training  systems,  there  is  no  way  to  integrate  the  AARs.  Furthermore,  the  same 
problem  is  present  for  single  echelon  training  on  two  different  training  systems  of  the 
same  simulation  category.  For  example,  two  platoons  performing  the  same  mission  on 
CCTT  and  SIMNET  will  each  receive  AAR  products  that  differ  in  format,  content,  and 
level  of  detail. 

None  of  the  BCTD  or  NSC  documents  explicitly  outline  a  goal  or  purpose  for 
AAR  aids  and  their  use.  (BCTD,  1996b;  BCTD,  1996a;  BCTD,  1995;  and  NSC,  1995) 
Given  the  literature  review  and  personal  experience,  AAR  products  should  guide/focus 
the  AAR,  efficiently  transmit  relevant  information  to  the  audience,  and  record  the  lessons 
learned.  Collectively,  the  products  should  show  the  critical  training  events,  results  of 
those  events,  and  the  logical,  cause-and-effect  linkage  between  training  event  and  lessons 
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learned.  Successful  distributed  training  not  only  requires  a  common  understanding  and 
perception  of  the  battlefield  amongst  linked  systems,  but  also  requires  a  common 
understanding  and  perception  of  the  feedback  resulting  from  the  training.  If  this 
understanding  and  perception  are  uncommon,  then  two  distributed  groups  may  not  have 
the  same  effectiveness  or  battlefield  situation  considerations  to  assess  and  improve 
performance.  However,  if  they  are  each  able  to  define  a  problem  and  identify  the  same 
factor  and  variable  relationships,  then  steps  to  improve  performance  can  be  orchestrated 
with  respect  to  their  units  and  each  other. 

Physically,  STAARS  exists  as  101  standard  information  formats  for  AAR 
products  that  have  been  developed  and  published  in  the  first  version  of  the  STAARS 
AAR  Handbook.  (BCTD,  1995)  There  are  five  standard  types  of  formats  that  STAARS 
lists:  the  battle  summary,  battle  set,  sketch,  statistical  report,  and  word  slide.  The  battle 
summary  is  a  video  animation  showing  friendly  and  enemy  force  maneuver.  The  battle 
set  is  a  single  frame  or  snapshot  from  the  battle  summary  animation.  The  sketch  is  a  user 
created  product  that  displays  a  concept,  an  aspect  of  the  battle,  or  doctrinal  principle.  The 
statistical  report  is  a  graph  or  chart  developed  from  exercise  data,  and  the  word  slide  is  a 
user  created  slide  of  written  text.  (BCTD,  1995) 

Surveying  the  content  or  these  product  formats  reveals  that  they  were  not 
developed  for  the  information  needs  of  small  unit  training  feedback.  Instead,  these 
products  focus  on  information  requirements  of  brigade  and  higher  level  units.  An 
example  of  this  shortcoming  can  be  seen  by  reviewing  the  required  maneuver  AAR 
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products  for  information  concerning  a  specific  engagement.  Given  that  a  platoon  and 
enemy  force  engage  in  a  fire  fight,  the  resulting  AAR  products  are  listed  in  Table  C-1, 
STAARS  AAR  Products  Supporting  Maneuver. 

Even  if  the  STAARS  could  track  and  report  individuals  and  vehicles  to  the  extent 
that  all  AAR  products  could  be  made  to  apply  to  all  echelons,  the  collection  of  products 
still  fails  to  show  which  individual  shot  another  individual  at  a  certain  location  and  point 
in  time.  The  platoon  and  company  AAR  products  must  portray  enemy  and  fiiendly 
actions/units  in  greater  levels  of  detail.  Specifically,  the  resolution  level  must  include 
individual  activities  and  communication  transmissions. 

In  fact,  the  AAR  products  are  more  suited  to  showing  unit  trends  over  an  extended 
period  of  time  or  multiple  engagements.  Each  is  designed  to  represent  an  aggregation  of 
units  or  statistics.  The  point  is  that  STAARS  does  not  support  training  below  brigade 
level.  Although  it  may  eventually  evolve  a  capability  to  do  so,  its  current  priority  is  to 
support  higher  echelon  training  and  R&D  efforts  first.  If  STAARS  is  to  be  the  Army’s 
baseline  AAR  specification,  a  thorough  analysis  to  define  the  standardized  platoon  and 
company  AAR  requirements  is  needed. 

At  small  unit  levels,  STAARS  does  not  capture  sufficient  information  for 
effective  performance  feedback  of  combat  operations/actions.  CCTT  AAR,  STRIPES 
and  ATAFS  all  have  the  capability  to  capture  information  pertaining  to  the  anatomy  of  a 
fire  fight.  Important  information  concerning  who  shot  first,  how  effective  was  the 
gunnery  of 
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Table  C-1,  STAARS  AAR  Products  Supporting  Maneuver 


AAR  Product 
(&  Type) 

Description 

Applicable 

Echelon 

Movement  Animation 
(Battle  Summary) 

series  animation  of  a  2-D  map  showing 
specific  units  moving  during  a  specified 
time  period 

All 

Maneuver  Forces; 
Available,  Committed, 
Engaged 

(Statistical  Report) 

line  graph  showing  units  available, 
committed,  and  engaged  in  the  area 

Company  - 
Division 

Maneuver  Battle  Sets 
(Battle  Set) 

two  dimensional  map  display  of  units  and 
fiieir  locations 

All 

Weighting  the  Main 
Effort 

(Statistical  Report) 

bar  graph  showing  the  distribution  of 
combat  power  between  the  main  effort  units 
and  supporting  effort  vmits 

Brigade 
and  higher 

Degradation  of  Forces 
(Statistical  Report) 

horizontal  line  chart  depicting  the  statistical 
degradation  of  friendly  and  enemy  forces 

All 

Movement  Time  Lines 
(Statistical  Report) 

horizontal  bar  graph  displaying  the  variance 
between  plaimed  and  actual  movement 
times 

Battalion 
and  higher 

Comparison  of  Forces  - 
Main  Effort 
(Statistical  Report) 

vertical  bar  graph  comparing  relative 
fidendly  and  opposing  force  levels  in  a 
specific  location  and  time  period  of  the 
battlefield 

All 

Use  of  Reserve 
(Sketch) 

timeline  showing  the  status  of  the 
designated  reserve  units 

Battalion  - 
Division 

Counterreconnaissance 
Effectiveness 
(Statistical  Report) 

vertical  bar  graph  showing  the  total  number 
of  OPFOR  reconnaissance  assets,  the 
number  detected,  and  the  number  destroyed 

All 

Fratricide 
(Battle  Set) 

two  dimensional  map  display  of  unit 
locations  and  the  firatricide  results 

All 

(BCTD,  1995, 21  through  36  and  114) 
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individual  vehicles,  when  did  the  enemy  see  the  Blue  force  and  when  did  the  Blue  force 
see  the  enemy  are  important  elements  that  need  to  be  discussed  during  an  AAR.  During 
live  training  at  the  CTCs,  this  level  of  detail  is  demanded  to  support  crew,  squad,  platoon, 
company,  and  battalion  AARs.  None  of  the  STAARS  products  require  or  present 
information  at  this  level  of  detail. 

The  most  efficient  and  effective  method  of  deciding  what  standard  format  should 
be  used  for  an  AAR  product  is  to  conduct  a  cognitive  task  analysis  (Appendix  A, 
Theoretical  Foundations,  and  Anderson,  1993)  of  what  the  trainee  must  perform.  To 
date,  this  has  not  been  done  to  the  required  level  of  detail.  The  level  of  detail  needed  is 
such  that  each  task  and  subtask  can  be  described  by  a  production  system  model.  This 
forces  the  identification  of  measurable  inputs  and  outputs  that  are  germane  to  the  task. 
These  are  the  inputs  and  outputs  that  then  need  to  be  reported  on  the  standardized  AAR 
aid.  Given  the  MTPs,  one  fourth  of  the  work  is  already  done.  The  MTPs  identify  the 
tasks  that  need  to  be  broken  into  primitive  subtasks  and  then  broken  into  declarative 
condition  chunks  and  production  rules.  Meliza  hints  at  this  but  does  not  elaborate  the 
need.  (Meliza,  1996a,  46)  This  would  eliminate  the  trial  and  error  efforts  of  documenting 
measures  of  performance  in  simulation  training  systems  (Meliza,  1996a,  54) 

An  additional  benefit  of  a  detailed  cognitive  task  analysis  is  that  the  AAR  aid  will 
directly  address  the  productions  and  declarative  misconceptions  that  caused  poor 
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performance.  These  are  exactly  the  elements  that  must  be  retrained.  Hence,  the  AAR 
becomes  a  planning  session  for  retraining  or  future  training.'*’ 

STAARS  is  not  scheduled  to  be  funded  until  the  1998  fiscal  year.  At  that  time, 
the  NSC,  the  BCTD,  Battle  Command  Training  Program  (BCTP),  and  Army  field 
commanders  will  evaluate  contract  proposals  for  the  system.  STAARS  will  then  be  built 
over  a  six  year  period  and  is  scheduled  for  fielding  in  2004.  (BCTD,  1996a) 


STMNET  and  UPAS 

As  stated  before,  SIMNET  was  purposefully  built  without  an  AAR  system. 
(Alluisi,  1991)  A  review  of  the  sources  that  recount  AAR  system  capabilities  and  history 
reveals  two  notes  concerning  the  absent  AAR  component  in  SIMNET:  first,  the  design  of 
AAR  systems  was  modular  so  that  diey  could  be  unplugged  from  the  training  system  and 
upgraded,  and  second,  SIMNET  became  a  test  vehicle  for  AAR  system  development. 
(Meliza,  1996,  Bessemer  et  al.,  1995,  Rankin  et  al.,  1995,  and  Meliza  et  al.,  1992b) 

The  modular  system  design  is  still  a  capability  that  is  sought  by  design  engineers. 
In  an  internal  memorandum  calling  for  a  collective  AAR  system  research  and 
development  effort,  the  Program  Manager  (PM)  of  CATT  emphasized  the  need  for  a 
common  DIS  compliant  AAR  system  with  an  “open  architecture  [so  that]  specific 
applications  could  be  ‘plugged  in’  to  the  core  [training]  system.”  (STRICOM,  1996d) 


The  term  “retraining”  denotes  the  training  that  occurs  immediately  after  a  training  exercise  to  correct 
deficiencies. 
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UPAS  was  the  initial  AAR  system  development  effort  for  SIMNET.  As  such,  it 
established  the  standard  of  needed  components  and  system  capabilities  for  subsequent 
AAR  systems.  It  is  a  personal  computer  (PC)  based  system  that  assists  trainers  and 
researchers  in  analyzing  unit  performance.  (Meliza  et  al.,  1992a,  1)  Thus,  UPAS  was  a 
tool  the  trainer  could  use  to  collect  and  present  information  for  the  AAR  or  training 
assessment.  At  the  same  time,  features  were  included  that  facilitated  scientific  research 
and  analysis.  (Meliza  et  al.,  1992a,  1)  Two  limitations  of  UPAS  are:  1)  all  exercise  data 
must  be  collected  before  its  analysis  capabilities  can  be  utilized,  and  2)  the  system  caimot 
monitor  radio  communications.  (Meliza  et  al.,  1992b,  27) 

The  components  of  UPAS  are:  the  data  collector  that  collects  all  pertinent  data 
broadcast  over  the  network;  PC  with  internal  tape  drive  and  printer  for  manipulation  of 
data  stored  on  a  relational  database;  and  large  television  monitor  to  display  exercise 
replay  segments  with  “stealth”  and  “out  the  window”  views.  Plan  View  Display  (PVD), 
Battle  Flow  Chart,  Battle  Snapshots,  Exercise  Timeline,  and  user  created  graphs  and 
tables.  (Meliza  et  al.,  1992b,  5, 6,  and  35  to  36,  and  Meliza  et  al.,  1992a,  1, 2,  and  A-1) 
upas’s  format  designs  of  the  PVD,  Battle  Flow  Chart,  Battle  Snapshots,  and 
Exercise  Timeline  AAR  aids  were  adapted  by  ATAFS  and  STAARS.  These  aids  were 
developed  and  tested  with  UPAS.  (Meliza  et  al.,  1992b)  The  PVD  is  a  two  dimensional 
map  view  of  a  specified  area  with  operational  graphics  and  imit  icons  superimposed  on  a 
map  grid  and  topographical  features.  The  PVD  has  adjustable  magnification  settings  so  a 
larger  or  smaller  area  can  be  displayed.  (Meliza  et  al.,  1992b,  22)  The  Battle  Flow  Chart 
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is  an  animated  replay  of  a  specified  area  and  time  period  that  shows  individiial  vehicle  as 
well  as  unit  movement,  relative  positioning,  operational  graphics,  map  grid  marks,  and 
topographical  features.  (Meliza  et  al.,  1992b,  23)  The  Battle  Snapshot  is  another  two 
dimensional,  animated  view  of  a  specified  area.  It  shows  an  instance  of  a  Battle  Flow 
Chart.  Hence,  previous  movement  and  direction  of  travel  are  not  readily  apparent  from 
this  aid.  (Meliza  et  al.,  1992b,  24)  The  fourth  aid  is  the  Exercise  Timeline.  It  is  a  “tool 
for  looking  at  temporal  coordination  of  movement,  control  measures,  and  firing  events.” 
(Meliza  et  al.,  1992b,  25) 

CCTT  AAR  System 

The  CCTT  AAR  workstation  consists  of:  a  PVD  that  can  display  both  two  and 
three  dimensional  replay  of  the  exercise  using  “stealth”  or  “out-the-window”  views  as 
well  as  display  graphs,  tables,  and  word  (text)  slides;  debriefing  display  that  is  a  color 
projection  screen  measuring  68  by  92  inches;  visual  display  that  provides  a  120  by  30.5 
degree  field  of  view;  control  console  to  log,  reduce,  and  analyze  all  network  data;  printer 
to  make  paper  copies  of  displayed  information;  radios  and  voice  recorder  to  collect  and 
record  all  voice  communications  traffic;  and  a  VHS  tape  recorder  to  record  specified 
exercise  video  clips  and  the  corresponding  voice  communications.  There  are  five 
workstations  to  accommodate  data  collection  of  five  simultaneous  and  separate  training 
exercises.  (LORAL/Army  IDT,  1996, 66  through  73, 167,  and  A-35  to  A-36) 
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An  attractive  feature  of  the  CCTT  system  is  the  capture  of  player  and  trainer  voice 
communications.  During  exercise  execution,  the  workstation  operator  can  communicate 
on  one  of  four  exercise  radio  channels  while  the  system  records  all  voice  communications 
on  all  exercise  channels.  Furthermore,  the  operator  can  make  time-stamped  voice 
armotations  on  a  separate  channel  during  replay.  (LORAL/Army  IDT,  1996, 72)  Thus, 
the  AAR  leader  can  aimotate  discussion  points  with  verbal  comments,  textual  comments, 
or  both. 

CCTT’s  AAR  console  has  preset  statistical  analysis  capabilities  that  may  be 
employed  after  the  exercise  when  all  the  data  is  collected.  The  workstation  operator  can 
choose  to  output  any  of  12  statistical  reports  once  the  analysis  is  complete.  However,  he 
may  not  customize  the  statistical  analysis  or  products.  (LORAL/Army  IDT,  1996, 71  and 
72) 

During  the  conduct  of  the  AAR,  the  system  requires  a  workstation  operator  to  run 
the  AAR  presentation.  The  operator  can  not  create  nor  use  just  any  combination  of  video, 
audio,  or  text  to  support  the  presentation.  (STRICOM,  1996d)  He  is  limited  to  what  the 
system  is  programmed  to  do. 

Currently,  the  take  home  package  that  CCTT  produces  consists  of  a  VHS  tape  of 
the  information  ported  to  the  debrief  screen.  (STRICOM,  1996d)  There  are  no 
mechanisms  to  record  the  audience’s  or  AAR  leader’s  comments/discussion  during  the 
AAR.  Planned  improvements  include  an  additional  VHS  recorder  per  workstation  to 
record  these  comments.  Other  forecasted  improvements  are  to  include  capabilities  that 
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will  support  the  production  of  a  video  and  paper  copy  take  home  package  that 
incorporates  media  and  data  from  all  or  any  of  the  training  exercises  conducted  during  a 
specified  time  period.  (STRICOM,  1996d) 

ATAFS 

The  ATAFS  is  a  workstation  consisting  of  a  35  inch  television  for  exercise  replay 
and  display  of  AAR  aids;  PC  with  external  tape  drive  and  printer  to  monitor  the  storage 
of  unit  activity  data,  VHS  video  recorder  to  record  the  AAR  presentation;  and  digitizing 
tablet  for  inputting  operational  graphics  prior  to  the  exercise.  (LB&M,  1996, 2)  There  are 
four  features  fiiat  set  ATAFS  apart  from  the  other  systems.  The  first  is  that  it  employs  an 
expert  knowledge  base  to  automatically  create  AAR  aids  that  are  very  similar  to  those  in 
UPAS.  The  second  feature  gives  the  user  the  capability  to  create  custom  AAR  aids 
before,  during,  or  after  the  exercise.  The  third  records  voice  communications  during  the 
exercise  and  can  synchronously  replay  both  video  and  voice  actions.  The  fourth 
capability  is  that  whatever  is  presented  on  the  large  TV  screen  during  the  AAR  is 
recorded  by  the  video  recorder  for  the  take  home  package. 

The  automatic  creation  of  AAR  aids  in  ATAFS  is  governed  by  an  expert  system 
ftiat  uses  tactical  events  as  condition  triggers  for  the  recording  of  predetermined  data 
(LB&M,  1996)  These  capabilities  reduce  the  AAR  leader/trainer  AAR  preparation 
workload.  The  ATAFS  automatically  generates  AAR  training  aids  and  discussion  points 
by  monitoring  and  analyzing  network  data  and  responding  to  0/C  interactive 
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commands/prompts.  Mission  based  rule  sets  control  the  automatic  generation  of  AAR 
products  for  generic  and  routine  combat  actions.  (LB&M,  1996)  This  frees  the  0/C  to 
monitor  the  critical  actions  that  are  unique  to  the  training  conditions  -  unique  specified 
and  implied  tasks,  the  enemy  array,  terrain  effects,  friendly  force  status,  and  time 
available.  Hence,  the  0/C  has  more  time  available  to  analyze  the  cause  and  effect 
relationship  between  an  action  and  results.  Current  improvements  being  developed  are  to 
expand  the  knowledge  base  to  monitor  more  collective  tasks  and  improve  the  system’s 
flexibility  and  responsiveness  to  both  AAR  preparation  and  exercise  control  use.  (Brown, 
1996) 

The  digitizing  tablet  and  work  station  allow  the  0/C  to  select  the  desired  AAR 
product  and  its  format  for  presentation.  (LB&M,  1996,  1-16)  The  ATAFS  format  types 
match  the  required  STAARS  product  format  types  as  shown  by  Table  C-2,  ATAFS  and 
STAARS  Product  Types.  Unlike  any  of  the  other  AAR  systems,  ATAFS  can  combine 
the  Word  Slide  aid  with  any  of  the  other  aids.  Hence,  ATAFS  does  have  some 
presentation  flexibility.  It  has  the  ability  to  capture  actions  and  sequences  of  actions  and 
present  them  in  any  of  its  available  formats.  The  O/C  can  select  or  reconfigure  the 
product  format  to  make  the  point  of  discussion  clear  to  the  audience.  However,  it  can  not 
employ  multimedia  presentation  methods. 
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Table  C-2,  ATAFS  and  STAARS  Product  Types 


ATAFS 

STAARS 

Plan  View  Animation  => 

Battle  Summary 

Snapshot,  Battle  Flow,  and  Fire  Fight  => 

Battle  Sets  and  Sketches 

Statistical  Aids  => 

Statistics  Reports 

Word  Slides  => 

Word  Slides 

As  with  the  other  systems,  the  presentation  medium  for  the  ATAFS  AAR  is  the 
TV  screen.  (LB&M,  1996,  1  through  25)  The  TV  screen  can  display  both  two  and  three 
dimensional  replay  of  the  exercise  using  “stealth”  or  “out-the-window”  views  as  well  as 
display  graphs,  tables,  and  word  (text)  slides.  Of  these  last  three  aids,  ATAFS  can 
display  a  combination.  Since  the  VCR  records  everything  on  the  TV,  the  AAR 
presentation  is  captured  and  ready  for  the  unit  at  the  end  of  the  AAR.  However, 
participant  comments  or  AAR  leader  remarks  are  not  automatically  included  in  the 
package. 

Essentially,  the  AAR  and  take  home  package  that  ATAFS  prepares  and  creates, 
respectively,  is  a  combination  of  AAR  leader  and  expert  system  lecture  notes  with 
supporting  slides  and  video  clips.  Both  are  void  of  participant  input  or  any  elaboration  of 
what  happened,  why  it  happened,  and  how  to  do  it  better.  ATAFS  does  give  the  AAR 
leader  more  time  to  think  about  and  prepare  the  AAR  by  automating  routine  tasks. 

Hence,  this  could  result  in  discussion  points  with  supporting  documentation  being  more 
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carefijlly  thought  out  and  presented.  ATAFS  does  help  the  AAR  leader  in  preparation  of 
the  AAR  more  than  the  other  systems  do. 


STRIPES 

STRIPES  consists  of  a  protocol  data  unit  (PDU)  scaimer  to  collect  data,  data 
logger  to  store  data,  and  a  two  dimensional  (2D)  plan  view  display  integrated  with  a  three 
dimensional  (3D)  stealth  view  display  for  exercise  replay.  (LORAL,  1994)  The  strongest 
feature  of  STRIPES  is  its  capability  to  archive  exercise  data.  An  Oracle  relational 
database  engine  allows  the  user  to  filter  data  and  perform  a  number  of  statistical  analyses. 
These  analyses  can  then  be  presented  via  graphs  and  tables  on  the  2D  display  or  printed 
to  paper.  (Hayes,  J.  [AcuSofl  Inc.],  personal  communication,  13  May  1996) 

Exercise  replay  is  accomplished  with  the  2D/3D  viewer  and  is  comparable  to  the 
replay  capabilities  of  CCTT  and  ATAFS.  Unlike  ATAFS,  the  preparation  of  AAR 
products  can  not  begin  imtil  the  exercise  data  is  logged.  However,  with  a  proficient 
workstation  operator,  STRIPES  has  the  capability  to  analyze  and  present  different  aspects 
of  the  exercise  data  during  the  AAR.  Hence,  the  operator  could  take  data  analysis 
commands  from  participants  during  the  AAR.  This  would  allow  clarification  of 
performance  outcomes  or  further  analysis  beyond  what  was  presented  by  the  AAR  leader. 

Like  the  AAR  aid  preparation,  take  home  package  generation  in  STRIPES  must 
occur  after  the  exercise.  The  database  engine  and  workstation  environment  does  speed 
the  process  of  collecting  and  combining  specific  data  elements  to  support  written 
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observations.  However,  tiiis  requires  a  skilled  database  operator  and  someone  to  identify 
which  data  elements  need  to  be  analyzed. 

STRIPES ’s  attractive  feature  is  its  ability  to  integrate  real  world  training  data  with 
the  training  execution  data  and  then  store  the  information  for  easy  statistical  analysis. 
While  this  feature  may  serve  operational  research  needs,  it  does  not  enhance  the  feedback 
given  the  trainee  immediately  following  the  training.  Potentially,  STRIPES  may  meet  the 
STAARS  requirement  to  digitize  the  training  exercise  for  storage  in  the  Army  Training 
Digital  Library  (ATDL).  (BCTD,  1995, 5  through  10)  However,  STAARS  has  not 
described  these  requirements  in  detail  and  future  technology  developments  may  meet  this 
need  more  efficiently  and/or  effectively. 


250 


APPENDIX  D 


IMPLEMENTATIONS  OF  INQUIRY  THEORY 
There  are  two  remarkable  demonstrations  of  the  effect  that  Inquiry  theory  has  on 
learning;  Valerie  J.  Shute’s  and  Robert  Glaser’s  Smithtown  (Shute  and  Glaser,  1990)  and 
Alan  Lesgold’s,  Gary  Eggan’s,  Sandra  Katz’s,  and  Govinda  Rao’s  Sherlock  (Towne  and 
Munro,  1991,  and  Lesgold  et  al.,  1992)  Both  of  these  are  simulation-based  systems 
designed  to  use  inquiry  strategies  for  domain  specific  knowledge  acquisition.  Each  found 
that  inquiry  skills  dramatically  increased  the  student’s  ability  to  remember  and  apply  the 
domain  knowledge. 

Smithtown 

Smithtown  is  an  intelligent  tutoring  system  that  systematically  guides  the 
student’s  discovery  of  microeconomics.  The  system  has  two  goals.  The  first  is  to  aid  the 
student  in  mastering  scientific  inquiry  skills  -  becoming  more  systematic  and  goal 
oriented  in  the  discovery  of  rules  and  theories.  The  second  is  to  impart  microeconomics 
subject  content  to  the  student.  (Shute  and  Glaser,  1990, 51  to  53)  Shute  and  Glaser  found 
that  the  “most  optimum  learner  behaviors  ...  are  systematic,  hypothesis-driven 
activities.”  (Shute  and  Glaser,  1990, 74)  In  other  words,  the  students  who  learned  the 
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most,  as  measured  by  pretest  -  posttest  change,  about  microeconomics  learned  and 
applied  inquiry  skills.  Relative  to  the  pretest,  those  who  did  poorly  on  the  posttest 
resisted  the  guidance  of  the  intelligent  tutor  in  Smithtown  or  were  in  the  control  group 
that  did  not  use  Smithtown.  (Shute  and  Glaser,  1990, 63) 

Shute  and  Glaser  identified  and  addressed  two  problems  when  developing  the 
instruction  tutor  for  induction  reasoning  and  hypothesis  testing.  These  problems  are  that 
“many  learners  can  induce  regularities/pattems  but  do  not  treat  them  as  hypotheses  to  be 
tested”  and  when  they  do  test  a  hypothesis,  many  use  faulty  methods  or  procedures  that 
do  not  guarantee  reasonable  or  relevant  conclusions.  (Shute  and  Glaser,  1990, 52)  These 
problems  were  the  impetus  for  the  goal  to  teach  effective  inquiry  skills. 

To  accomplish  the  goal,  Smithtown  constantly  maintains  and  monitors  a  model  of 
student  actions.  The  system  is  able  to  determine  student  process  errors  by  comparing 
student  actions  and  sequences  of  actions  to  an  expert  model  of  inquiry  rules/theories. 

This  expert  model  is  stored  in  a  procedural  knowledge  base  of  inquiry  productions. 

(Shute  and  Glaser,  1990, 53,  and  Williams,  1996a)  The  system  makes  the  student  aware 
of  the  elements  of  inquiry  skill  and  then  allows  him  to  learn  them  through  practice.  For 
example,  if  the  student  conducts  an  experiment  to  determine  what  factors  affect  a 
dependent  variable  and  chooses  to  vary  two  or  more  factors  without  reviewing  baseline 
data,  then  the  tutor  recommends  that  he  review  the  data  and  vary  only  one  factor  at  a 
time.  (Shute  and  Glaser,  1990,  55)  Hence,  the  student’s  application  of  inquiry  methods 
are  diagnosed. 
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Students  manipulate  variables,  observe  effects,  and  organize  information  to 
explore  the  microeconomics  principles  embedded  in  Smithtown.  (Shute  and  Glaser,  1990, 
56)  The  system  prompts  the  student  in  two  types  of  systematic  investigation:  “1) 
explorations  -  observing  and  obtaining  information  to  generate  hypotheses  about  the 
mieroeconomics  concepts  and  laws;  and  2)  experiments  -  a  series  of  student  actions 
conducted  to  confirm  or  differentiate  h5^otheses.”  (Shute  and  Glaser,  1990, 56) 

Smithtown  also  diagnoses  the  quality  of  the  student’s  understanding  of  the 
domain  knowledge.  The  system  stores  the  economics  factors  and  their  relationships  in  a 
declarative  knowledge  base.  By  requiring  the  student  to  adhere  to  a  specific  interaction 
format,  the  tutor  can  relate  student  input  to  factor  relationships  of  supply  and  demand. 

The  student’s  generalized  rule/theory  can  be  assessed  for  completeness  by  comparison  to 
the  domain  knowledge  base.  For  example,  if  the  student  makes  a  correct  hypothesis 
about  which  factors  affect  a  dependent  variable,  then  the  system  acknowledges  the 
discovery  and  elaborates  the  rule/theory  further  to  the  student.  (Shute  and  Glaser,  1990, 
53) 

Smithtown  is  remarkable  for  instruction  efficiency.  The  performance  difference 
between  students  instructed  by  Smithtown  for  5  hours  and  students  instructed  by 
university  economics  professors  for  1 1  hours  were  not  statistically  significant.  (Shute  and 
Glaser,  1990, 67  and  73)  With  less  than  half  the  instruction  time  of  students  in  an 
undergraduate  economics  course,  students  using  Smithtown  performed  at  a  comparable 
level  on  economics  tests. 
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In  the  Smithtown  experiments,  Shute  and  Glaser  were  able  to  determine  what 
made  students  more  or  less  successful.  The  result  was  a  list  of  the  behaviors  exhibited  by 
successful  students,  as  determined  by  pretest-posttest  change,  that  had  the  most 
significant  statistical  effect.  These  behaviors  are  listed  in  order  of  significance  in  Table 
D-1,  Significant  Inquiry  Behaviors.  Also  listed  in  the  table,  are  Collins’  Inquiry  theory 
strategies  that  correspond  to  the  significant  behaviors  of  Smithtown  students. 

With  respect  to  Collins’  strategies,  Smithtown  is  predicated  on  and  implements 
the  questioning  authority  strategy  (Chapter  IV,  pg.  1 65).  By  not  allowing  the  student  to 
rely  upon  the  teacher  or  intelligent  tutor  to  supply  the  answer,  it  forces  the  student  to 
think  and  experiment  on  his  own.  It  rewards  logical  sequences  of  behaviors  and 
highlights  and  corrects  illogical  behaviors. 

The  intelligent  tutor  in  Smithtown  reinforces  student  behavior  that  corresponds  to 
Collins’  strategies  that  the  student  uses  to  acquire  domain  knowledge.  Specifically,  it 
requires  students  to:  form  hypotheses,  generate  hypotheses,  test  hypotheses,  vary  cases 
systematically,  and  consider  alternative  predictions  (Table  D-1,  Significant  Inquiry 
Behaviors  in  Smithtown). 
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Table  D-1,  Significant  Inquiry  Behaviors  in  Smithtown 


Corresponding 

Significant  Description  Inquiry 

Behavior  Theory  Strategies 


1.  Generalization  Testing  developing  economic  beliefs  across 
markets  -  generalizing  a  concept  across 
related  and  unrelated  goods. 


2.  Complexity  of 
experiment 


Number  of  times  a  specific  factor  was  varied 
and  the  average  number  of  connected 
actions  associated  with  an  experiment.''* 


3.  Systematic  Number  of  variables  changed  per  experiment 
variable  iteration  (step)  to  discover 

changes  interrelationships. 


4.  Adequate  data 
collection 


Number  of  times  sufficient  data  was  gathered 
before  generalization. 


5.  Planning  an  Number  of  times  connected  actions  occurred 
experiment  that  had  been  specified  (planned) 

beforehand  -  planned  variable  changes  that 
were  performed. 


6.  Predicting  Number  of  specific  predictions  made  divided 
experimental  by  the  number  of  general  hypotheses  made, 
outcomes 


7.  Notebook 
entries 


Number  of  relevant  notebook  entries  divided 
by  the  total  number  of  notebook  entries.^® 


Generating 

hypotheses 

Testing  hypotheses 

Considering 

alternative 

predictions 

Varying  cases 
systematically 

=>  Testing 
hypotheses 

=>  Forming 
hypotheses 

[feedback  measure 
not  accounted  for 
by  Collins] 


Forming  hypotheses 


[feedback  measure 
not  accounted  for 
by  Collins] 


““  Reference  behavior  #  2;  connected  actions  are  actions  associated  with  defining  a  set  of  factors  and  dieir 
interrelationships. 

Reference  behavior  #  7;  relevant  notebook  entries  refers  to  the  entries  associated  with  the  variables 
specified  in  the  plan  of  the  experiment.  Overall,  successful  students  made  more  notebook  entries  than  less 
successful  students.  Additionally,  those  entries  were  relevant  to  the  focus  of  their  investigation.  (Shute  and 
Glaser,  1990, 67)  This  confirms/supports  behavior  #  4,  gathering  adequate  data  before  hypothesizing. 
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Sherlock 


With  Sherlock,  Lesgold  et  al.  establish  a  coached  practice  environment  to  train 
diagnosis  and  repair  of  the  F-1 5  Manual  Avionics  Test  Station.  The  test  station  itself  is 
used  to  diagnose  failures  of  navigational  equipment  on  the  F-1 5  aircraft.  However,  when 
the  test  station  fails,  it  must  be  diagnosed  and  repaired  by  a  human  technician.  The 
system’s  goal  is  the  development  and  refinement  of  mental  models  to  enhance  the 
student’s  problem  solving  and  inductive  learning  skills.  (Lesgold  et  al.,  1992,  51,  and 
Towne  and  Munro,  1991, 328) 

Lesgold  et  al.  integrate  intelligent  simulation  and  training  technology  with 
principles  of  apprenticeship.  These  principles  are  that:  students  learn  by  doing,  the 
acquired  knowledge  is  anchored  in  experience,  and  skill  development  is  supervised  by  a 
tutor  with  expert  knowledge.  Simulation  and  training  technology  are  used  to  allow  the 
trainee  to  perform  the  task  he  is  trying  to  learn.  With  respect  to  just  having  exercises  to 
practice  in  a  classroom,  students  are  more  motivated  to  learn  when  they  have  physical 
tasks  to  accomplish  with  actual  tools.  Additionally,  this  hands-on  application  “anchors” 
the  knowledge  that  is  acquired  in  personal  experience.  In  turn,  this  enhances  recall  of  the 
knowledge  chunks  and  subsequent  performance.  (Lesgold  et  al.,  1992, 50) 

An  intelligent  system  is  used  to  provide  the  trainee  expert  support  and  coaching. 
Specifically,  the  tutor  assists  “in  the  delicate  period  during  which  a  trainee  knows  more  or 
less  what  to  do  in  a  problem  situation  but  is  unable  to  keep  track  of  his  efforts  because 
each  requires  focused  attention  to  be  carried  out  successfully,  coworkers  and  ‘masters’ 
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become  supportive  external  memories  that  share  the  attentional  load  while  still  affording 
opportunities  for  practice.”  (Lesgold  et  al.,  1992, 49  through  50)  The  result  is  a  “situated 
learning  approach  that  fosters  conceptual  abstraction  via  coaching  and  comment  upon 
specific  problem  situations.”  (Lesgold  et  al.,  1992,  57) 

Sherlock’s  intelligent  tutor  coaches  the  student’s  problem-solving  performance, 
assesses  student  strengths  and  weaknesses  through  model  tracing,  and  assigns  progression 
problems  based  upon  the  assessments.  (Lesgold  et  al.,  1992, 51)  An  analysis  of  this 
system  reveals  that  the  problem  solving  strategies  are  an  adaptation  of  Inquiry  theory. 

The  students  that  train  with  Sherlock  have  a  basic  working  knowledge  of  electrical 
circuitry.  Thus,  the  students  are  not  novices  and  know  most  of  the  factors  that  may  cause, 
or  combine  to  cause,  a  system  failure.  Sherlock  teaches  the  student  to  efficiently  find  and 
diagnose  the  failure.  In  terms  of  Collins’  Inquiry  theory,  it  accomplishes  this  by 
requiring  the  student  to  form  a  hypothesis  about  the  location  of  the  failure,  test  the 
hypothesis,  and  repeat  the  steps  until  the  location  is  determined.  The  student  can  consult 
the  tutor  to  debug  his  hypothesis  by  tracing  consequences  to  a  contradiction,  considering 
alternative  predictions,  varying  cases  systematically,  and  establishing  {selecting)  positive 
and  negative  exemplars.  Lesgold  et  al.  refer  to  this  interaction  with  the  tutor  as 
“reflective  follow-up.”  (Lesgold  et  al.,  1992, 57  through  58)  This  process  is  repeated  for 
diagnosing  the  cause  of  circuit 
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failure  once  the  location  is  determined.  By  employing  Inquiry  strategies,  the  student  is 
able  to  realize  the  factors  that  indicate  the  location  and  nature  of  a  failure  in  the  test 
station. 


Sherlock’s  tutor  monitors  five  basic  student  behaviors.  These  behaviors  are  listed 
in  Table  D-2,  Student  Inquiry  Behaviors  Monitored  in  Sherlock. 


Table  D-2,  Student  Inquiry  Behaviors  Monitored  in  Sherlock 


Behavior 

Description 

Corresponding  Inquiry  Strategy 

Swapping  vs. 
testing 

Student  does  not  swap  a 
component  luitil  he  has 
proven  that  it  is  defective. 

Testing  hypotheses 

Varying  cases  systematically 

Redundant 

testing 

Student  systematically  tests 
components  such  that  none 
must  be  tested  twice. 

Testing  hypotheses 

Varying  cases  systematically 

Accepting 

help 

Student  asks  for  help  when 
needed.  He  compares  his 
approach  or  solution  to  that 
of  the  tutor. 

Considering  alternative  predictions 
Tracing  consequences  to  a 
contradiction 

Selecting  positive/negative 
exemplars 

Independence 
and  self- 
confidence 

Student  does  not  ask  for  help 
when  not  needed. 

None 

Systemticity 

Testing  efficiency;  student 
restricts  the  fault  search  to 
the  relevant  circuit  path. 

Testing  hypotheses 

Lesgold  et  al.  have  developed  a  second  generation  system,  Sherlock  II,  that  has 
not  yet  been  tested  with  students.  However,  the  first  version,  Sherlock,  has  been  tested 
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for  its  effect  on  learning.  The  effect  Sherlock  had  on  training  was  determined  by 
measuring  pretest— posttest  change  and  is  even  more  remarkable  than  that  of  Smithtown. 
Lesgold  et  al.  found  that  20  to  25  hours  of  Sherlock  practice  time  produced  average 
performance  improvements  that  were  commensurate  with  the  effects  of  4  years  job 
experience.  (Lesgold  et  al.,  1992,  54)  While  these  results  seem  incredible,  the  learning 
gain  from  Sherlock’s  approach  was  independently  verified  with  32  college  students. 
Johnson  et  al.  (1993)  found  that  the  tutorial  group  showed  a  78%  improvement  (in  actual 
troubleshooting  success)  over  the  control  group. 

Although  testing  a  large  number  of  circuit  boards  seems  like  a  mimdane,  time 
consuming  task,  it  is  a  relevant  example  for  an  AAR.  The  location  and  correction  of 
faults  in  the  test  set  requires  the  knowledge  of  a  number  of  different  factors  and  their 
relationships.  While  the  actual  testing  methods  (using  an  oscilloscope  or  multimeter)  are 
simple  tasks,  the  student  must:  1)  consider  what  the  test  station  was  trying  to  do  when  it 
failed,  2)  develop  a  mental  representation  of  the  circuitry  involved  in  the  failed  function, 
and  3)  develop  a  plan  for  testing  that  functional  circuit.  Otherwise,  the  student  is  left 
searching  tens  of  thousands  of  parts  for  the  fault.  (Lesgold  et  al.,  1992,  51  and  52) 
Essentially,  the  student  must  learn  the  cause  and  effect  relationships  between  components 
and  die  troubleshooting  methodology  used  by  the  system  to  narrow  the  choices  for  the 
fault  location.  Then,  the  student  must  develop  an  efficient  circuit  test  plan  to  verify  his 
hypothesis.  The  better  he  understands  the  factor  relationships,  the  sooner  he  will  isolate 
and  correct  the  problem. 
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Like  the  airmen  trained  on  Sherlock,  AAR  participants  have  different  levels  of 
expertise  with  differing  levels  of  realization  about  factor  relationships.  Collectively,  the 
unit  is  not  at  the  novice  level.  What  Sherlock  has  shown  is  that  immense  performance 
improvement  can  be  made  by  focusing  the  students  on  the  problem  and  aiding  them  in 
discovery  of  the  factor  relationships  specific  to  that  problem. 

Self-elaboration  is  considered  to  have  a  strong  positive  impact  on  learning  even 
though  its  effect  has  never  been  empirically  isolated  in  that  respect.  In  previous  learning 
experiments,  elaboration  has  always  been  confovmded  with  one  or  more  factors. 
(Williams,  1996a)  Notwithstanding,  ACT-R  (Anderson,  1993)  and  the  general  theory  of 
problem  solving  (Simon  &  Newell,  1972)  provide  a  clear  and  plausible  theoretical  basis 
for  the  premise  that  one  learns  by  analogy  via  elaboration.  Both  Smithtown  and  Sherlock 
demonstrate  and  support  this  premise.  Both  systems  employ  versions  of  Inquiry  theory  to 
explain  learning  points  to  the  student,  keep  the  student  focused  on  the  learning  point  until 
he  understands  it,  and  then  require  him  to  apply  or  demonstrate  the  learning  point.  The 
mechanism  that  allows  the  student  to  imderstand  the  point  is  iterative  elaboration  of 
experience.  In  the  end,  self-elaboration  becomes  synonymous  with  mental  practice. 
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APPENDIX  E 


PROPOSED  DESIGN 

The  proposed  design  builds  upon  existing  AAR  systems  and  efforts  that  support 
small  unit  training  systems.  Previous  simulation  training  AAR  systems,  as  well  as  R  &  D 
efforts,  have  the  goals  of  diorough  performance  data  collection  and  the  facilitation  of  data 
manipulation  for  ease  of  preparation  and  presentation.  (Chapter  III  and  Appendix  C, 
Current  AAR  Systems  in  Training  Simulations)  This  design  does  not  discount  these 
efforts  but  describes  the  components  and  processes  needed  to  conduct  an  effective  AAR. 
Where  current  AAR  systems’  functionality  stops  with  the  display  of  performance  data, 
the  proposed  system  requires  participants  to  interactively  manipulate  the  data  during  the 
AAR.  Trainees  interact  with  each  other  and  manipulate  the  performance  data  in 
accordance  with  the  Inquiry  process  presented  in  Chapter  IV.  Figure  E-1  shows  the 
physical  components  of  the  proposed  AAR  design. 

Component  Functionality 

Control  Station 

The  control  station  is  the  AAR  leader’s  interface  with  the  data  collection 
subsystem.  Inquiry  tutor,  and  testing  component.  It  is  a  menu  driven,  graphical  user 
interface  (GUI)  that  allows  the  leader  to  input:  data  and  commands  into  the  collection 
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Figure  E-1,  Proposed  AAR  System  Design 

subsystem.  Inquiry  strategies  into  the  Inquiry  tutor,  and  test  parameters  into  the  testing 
component.  Also,  the  leader  controls  the  2D  and  3D  displays  from  the  workstation. 

The  AAR  leader  works  through  a  workstation  operator  who  can  devote  full  attention  to 
the  GUI  and  the  creation  of  the  needed  displays  during  the  AAR.  The  operator  is  needed 
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because  1)  four  different  components  must  be  controlled,  two  of  which  the  trainees  must 
interact  with  during  the  AAR;  and  2)  the  AAR  leader’s  primary  responsibility  is  to 
monitor  the  discussion,  from  which  interaction  with  a  GUI  will  detract. 

The  workstation  must  allow  the  AAR  leader  to  control  the  data  collection 
subsystem  and  the  Inquiry  tutor  during  the  AAR  in  response  to  discussion  queries  and 
clarification  needs;  for  example,  what  happened  at  a  specific  time  and  location  in  the 
training  exercise.  The  AAR  leader  must  also  have  the  capability  to  control  the  testing 
component  and  its  display  output  from  the  workstation.  These  requirements  mean  that 
workstation  and  AAR  system  responses  are  in  real  time  and  do  not  delay  the  AAR 
discussion.  Exercise  replay  and  test  control  must  include  the  capabilities  to  fast-forward, 
reverse,  pause,  advance/reverse  by  time  step,  and  advance/reverse  by  event. 

Data  Collection  Subsystem 

The  Data  Collection  Subsystem  component  must  be  STAARS  compliant  because 
it  will  be  implemented  by  the  year  2004.  (BCTD,  1996a)  Notwithstanding,  the  proposed 
AAR  system  is  not  dependent  upon  any  one  of  the  exercise  data  collection  systems 
(CCTT,  UPAS,  ATAFS,  STRIPES)  that  were  reviewed  in  Chapter  III  and  Appendix  C. 

Whatever  form  this  subsystem  takes,  the  AAR  leader  must  be  able  to  present  a 
selection  of  problem  issues  as  well  as  potential  dependent  variables  to  the  trainees.  To  do 
this,  the  collection  subsystem  must  collect  and  format  performance  data  associated  with 
collective  and  individual  task  standards.  Other  sources  that  help  to  determine  what 
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problem  to  solve  during  the  AAR  are  the  training  plan  (objectives),  the  senior  trainer,  OC 
observations,  and  OPFOR  observations. 

As  long  as  the  data  collection  subsystem  is  STAARS  compliant,  the  system  will 
be  able  to  collect  performance  data,  make  it  available  to  an  operator  for  manipulation,  and 
then  arrange  it  into  one  of  the  standard  formats.  For  example,  if  the  training  objective  is 
to  perform  the  collective  task  Perform  Overwatch/Support  by  Fire  to  the  standards 
outlined  in  ARTEP  7-8-MTP  (DA,  1994),  then  the  trainees  may  want  to  review  the  data 
associated  with  the  task  standard:  “The  platoon  delivers  suppressive  fires  to  prevent 
enemy  direct  fires  from  fixing  the  movement  element”  (DA,  1994, 5-18);  assuming  this 
standard  was  not  met.  The  data  collection  subsystem  would  then  need  to  provide  a 
sequence  of  time-stepped  battle  sets  (Table  C-1,  STAARS  AAR  Products  Supporting 
Maneuver)  depicting  the  overwatch  element,  the  maneuver  element,  and  the  enemy  forces 
during  the  period  of  the  assault  or  maneuver  bound.  As  each  battle  set  is  reviewed,  the 
enemy  positions  that  place  direct  fire  on  the  fiiendly  force  and  the  casualties  that  result 
should  be  highlighted  by  the  AAR  leader  or  trainee  imit  leader.  The  discussion  then 
begins  on  how  the  support  element  can  suppress  the  enemy  that  caused  the  fiiendly 
casualties. 

As  one  can  see,  the  battle  sets  must  be  readily  available  for  presentation.  If  the 
training  objective  is  used  to  guide  AAR  preparation,  then  this  data  and  format  will  not  be 
a  problem.  If  the  data  collection  subsystem  is  STAARS  compliant,  the  battle  sets  will 
either  be  automatically  generated  or  readily  available  for  generation. 
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Presentation  Medium 


The  critical  requirement  for  the  presentation  medium  is  that  it  be  clearly  seen  by 
all  participants  during  the  AAR.  The  AAR  presentation  medium  for  existing  AAR 
systems  and  systems  under  development  is  a  large  cathode-ray  tube  (CRT)  monitor  -  a 
large  screen  television.  (Appendix  C,  Current  AAR  Systems  in  Training  Simulations) 

The  proposed  system  employs  this  same  screen  for  the  same  purposes:  to  present  2D  and 
3D  animated  exercise  replay,  battle  sets  and  sketches,  statistical  summaries,  word  slides, 
and  any  other  standard  format  designated  by  STAARS.  (Appendix  C,  Current  AAR 
Systems  in  Training  Simulations) 

The  medium  must  be  clearly  seen  by  all  the  AAR  participants  seated  around  a 
terrain  model.  (See  Figure  E-2,  Physical  Layout)  With  groups  as  large  as  40  ,  there  will 
need  to  be  at  least  two  screens  to  accommodate  everyone.  A  single  screen  large  enough 
to  be  clearly  seen  across  a  room  will  be  too  large  to  view  comfortably  from  the  near  side 
of  the  room. 

An  alternative  to  the  two  screen  option  is  a  movie  theater  approach.  This  places 
the  viewing  screen  at  a  distance  from  the  audience  while  allowing  comfortable  viewing 
from  both  near  and  far  positions.  This  means  that  the  room  must  be  enlarged  and  the 
projection  system  changed  for  all  current  AAR  systems  as  well  as  those  in  development. 
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Discussion  Mediums 


Discussion  mediums  are  tools  that  facilitate  trainee  and  AAR  leader  articulation 
and  elaboration  during  discussion.  There  are  two  principal  mediums  in  the  proposed 
AAR  system,  the  terrain  model  and  the  whiteboard/chalk  board.  Each  must  be  located  so 
that  all  participants  can  see  and  have  access  to  them.  This  means  that  the  participants  are 
positioned  around  the  terrain  model  with  the  whiteboards  positioned  around  them.  (See 
Figure  E-2,  Physical  Layout) 


Figure  E-2  ,  Physical  Layout 


The  central  location  of  the  terrain  model  is  necessary  so  that  it  becomes  the  focus  of 
discussion.  In  this  arrangement,  it  is  more  difficult  for  a  single  person  to  lecture  to  the 
group  because  he  is  not  positioned  to  monopolize  the  group’s  attention. 


The  terrain  model  is  a  scaled  model  of  the  battlefield  where  the  action  being 
discussed  took  place.  Weapons,  equipment,  vehicles,  units,  and  individual  people  can  be 
represented  with  corresponding  scale  models.  However,  the  scales  must  be  large  enough 
for  all  the  participants  to  distinguish  individual  piece  characteristics.  For  example,  the 
trainees  must  be  able  to  visually  differentiate  between  types  of  vehicles  and  dismounted 
soldiers. 

A  3D  representation  is  needed  to  accurately  model  entity  time-distance 
relationships  and  the  physical  features  of  the  terrain.  The  entity  time-distance 
relationships  allow  the  trainees  to  quickly  visualize  the  relative  force  dispositions.  These 
relationships  are  constantly  assessed  against  an  analysis  of  the  terrain.  The  terrain 
representation  allows  the  trainees  to  uniformly  interpret  the  terrain  cues  and 
considerations.  Accurate  assessment  and  advantageous  use  of  cover  and  concealment 
offered  by  the  terrain  are  often  major  factors  in  a  fire  fight. 

Because  the  discussion  will  involve  different  actions  at  different  times,  this 
medium  must  have  the  capability  to  represent  a  number  of  locations  and  unit  situations  - 
die  terrain  model  must  be  re-configurable  in  real  time.  Furthermore,  the  re-configuration 
of  the  terrain  model  should  not  take  longer  than  one  minute  in  order  to  help  participants 
demonstrate  discussion  points  with  the  medium.  Since  a  permanent  terrain  model  cannot 
be  reconfigured  to  represent  different  locations  of  the  battlefield  and  a  sand  table  would 
take  too  long  to  make,  a  technological  solution  is  needed.  These  requirements  are 
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satisfied  by  a  new  display  technology  known  as  Laser-Based  3  Dimensional  (3D) 
Volumetric  Display  System.  (Soltan  et  al.,  1995) 

The  Laser-Based  3D  Volumetric  Display  System  was  developed  by  Parvis  Soltan 
xmder  a  contract  from  the  Naval  Command,  Control  and  Surveillance  Center,  The  system 
allows  a  person  to  view,  with  the  naked  eye,  color  3D  images  that  are  updated  in  real  time 
to  create  a  scene  in  a  volume  image  space.  A  cylindrical  volume  image  is  created  by 
incorporating  a  36-inch  diameter  double  helix  that  spins  at  10  revolutions  per  second.  A 
computer  controlled  laser  beam  illuminates  discrete  volume  points  (voxels)  on  the  helix. 
The  voxel  is  the  3D  counterpart  of  the  pixel.  The  helix  scatters  the  laser  light  so  that  each 
voxel  appears  to  emanate  from  a  specific  point  in  space.  To  create  a  scene,  up  to  40k 
laser-generated  voxels  are  projected  onto  the  reflective  surface  of  the  rotating  helix  using 
the  Acousto-Optic  Random- Access  Scanner.  These  voxels  are  refreshed  at  a  rate  of  20 
Hz  per  color.  Each  point  in  the  cylindrical  volume  has  an  x,  y,  z  coordinate  address 
stored  in  the  computer.  The  computer  then  synchronizes  the  laser  beam,  the  Acousto- 
Optic  (AO)  Scanner,  and  the  phase  of  the  helix  to  illuminate  a  set  of  points  and  create  an 
image.  (Soltan,  1995, 1)  The  capability  to  display  3D  images  that  can  be  viewed  from 
any  side  of  the  display  volume  makes  it  ideal  for  exercise  replay  and  testing  component 
display. 

An  alternative  to  the  Laser-Based  3D  Volumetric  Display  System  is  the  Mirage. 
The  Mirage  is  a  3D,  stereoscopic,  pseudo-holographic  display  system  built  by  Scott 
Smith  of  the  Institute  for  Simulation  and  Training  (1ST),  University  of  Central  Florida 
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under  a  contract  funded  by  STRICOM,  the  Defense  Modeling  and  Simulation  Office,  the 
Army  Research  Institute,  and  the  University  of  Central  Florida.  (Smith  &  Gamsey,  1997) 
The  term  “pseudo-holographic”  as  a  descriptor  refers  to  the  hologram-like  image  the 
system  produces.  It  does  not  use  true  holography.  Overall,  the  system  projects  a  3D 
scale  model  image  onto  a  horizontal  projection  table.  This  allows  the  viewer  to  walk 
aroxmd  the  image  and  zoom  in  and  out.  A  hand-held  joystick  allows  the  viewer  to  pan 
through  adjacent  scenes  in  the  database. 

The  Mirage  receives  scene  database  PDUs  via  both  Ethernet  and  FDDI  network 
interfaces.  The  system  consists  of  an  image  generator  that  provides  stereo  output  at  60 
right  and  60  left  eye  images  per  second  to  an  Electrochrome  Marquee*”  8110  series 
projector.  A  trapezoidal  mirror  bends  the  projected  image  to  a  vertical  path  onto  the 
projector  screen  laid  on  a  horizontal  table.  The  viewer  wears  StereoGraphics  Corporation 
CrystalEyes  2*”  glasses  to  see  the  time-multiplexed  right  and  left  eye  images  projected 
upon  the  screen.  The  glasses  are  field  sequential  electro-stereoscopic  liquid  crystal 
shutter  devices  synchronized  by  an  extended  range  infrared  transmitter  mounted  above 
the  table.  The  IR  transmitter  receives  its  signals  from  a  serial  port  on  the  image 
generator.  When  a  right  eye  image  is  being  drawn,  the  glasses  occult  the  left  eye  and  vice 
versa.  The  viewer’s  position,  relative  to  the  table  screen  is  determined  by  an  Ascenscion 
Technology  Corporation  Flock  of  Birds'”  magnetic  position  sensor  and  transmitter.  The 
sensor  (receiver)  is  worn  on  the  viewer’s  head  while  the  transmitter  is  mounted  beneath 
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the  center  of  the  table  screen.  The  viewer  controls  the  display  through  a  keyboard, 
mouse,  and  SpaceBall*™  joystick. 

The  principal  drawback  of  the  Mirage  system  is  that  it  will  only  allow  a  maximum 
of  two  viewers  per  table  screen.  These  viewers  must  be  within  a  10  degree  viewing  angle 
of  each  other  to  avoid  distortion  of  the  image.  A  single  viewpoint  can  be  transmitted  to  a 
number  of  glasses,  but  the  networked  viewers  are  at  the  mercy  of  the  viewer  with  the 
position  sensor.  His  head  movements  will  most  likely  give  the  rest  motion  sickness. 

Whiteboards  with  color  pens  are  for  participants  to  sketch  concepts,  take  notes, 
and  communicate  ideas  during  discussion.  The  boards  are  equipped  with  electronic 
scanners  so  that  their  contents  can  be  transferred  to  paper  copy  as  needed.  This 
component  records  the  written  discussion  notes  for  the  imit’s  take  home  package  and 
reduces  trainee  discussion  inattention  caused  by  trying  to  copy  everything  from  the  board 
to  a  notebook.  In  most  of  the  AARs  surveyed  in  Chapter  III,  a  unit  member  was 
employed  to  take  notes  on  the  AAR  leader’s  whiteboard  during  the  discussion. 

Drawing  pictures  helps  participants  articulate  their  argument  and  helps  others 
visualize  and  understand  the  idea.  The  medium  is  also  used  to  annotate  discussion  notes. 
In  this  case,  it  works  as  a  working  memory  aid.  As  participants  brainstorm,  the  ideas  are 
written  on  the  board  for  later  assessment.  Key  principles  and  relationships  are  written 
and  updated  on  the  board  so  that  participants  can  concentrate  on  deriving  or 
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understanding  new  rules  and  do  not  have  to  remember  ever5dhmg  that  was  stated 
previously. 

The  whiteboard  medium  is  particularly  useful  in  actively  engaging  trainees  in  the 
discussion  to  achieve  a  collaborative  effort.  By  making  reluctant  unit  members  move 
from  a  passive  sitting  position  to  writing  on  a  board,  they  are  induced  to  take  a  more 
physically  active  role  in  the  AAR.  This  activity  is  the  first  step  toward  constructively 
contributing  to  the  discussion. 

Inquiry  Tutor 

The  Inquiry  tutor  component  is  the  expert  knowledge  base  for  Inquiry  theory. 

The  tutor  maintains  Collins’  Inquiry  strategies  and  procedures  in  a  knowledge  base.  This 
procedural  knowledge  base  monitors  a  model  of  trainee  problem  solving  actions  in  the 
same  maimer  as  Shute  and  Glaser’s  Smithtown  (Appendix  D).  From  a  comparison  of 
expert  actions  and  trainee  actions,  the  tutor  offers  guidance  and  coaching  of  the  Inquiry 
approach  for  AAR  participants. 

The  tutor  keeps  a  detailed  record  of  all  trainee  inquiry  actions  and  categorizes 
them  as  either  behaviors  or  solutions.  The  tutor  then  diagnoses  solutions  by  comparing 
the  trainees’  solutions  to  sets  of  actions  or  non-actions  stored  in  the  procedural 
knowledge  base.  These  sets  constitute  optimal  and  sub-optimal  behaviors.  The  sets  of 
behaviors  are  essentially  proper  and  improper  sequences  of  Inquiry  actions.  The 
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differences  between  the  two  models  form  the  basis  of  the  Inquiry  approach  coaching  for 
the  group.  (Shute  and  Glaser,  1990, 53  and  54) 

The  Inquiry  tutor  will  rely  on  the  AAR  leader  to  input  the  trainees’  Inquiry 
actions  at  the  control  workstation.  The  tutor’s  coaching  will  then  be  displayed  on  the 
presentation  medium  to  guide  the  discussion. 

Testing  Component 

The  testing  component  of  the  proposed  AAR  system  is  used  to  test  trainees’ 
hypotheses  about  factors  that  affect  performance  outcomes.  The  testing  component  must 
be  a  constructive  simulation  of  situational  exercises  devised  by  the  trainees  and  capable 
of  faster  than  real  time  simulation.  This  speed  is  necessary  to  prevent  the  running  of  the 
simulation  from  distracting  and  delaying  the  AAR  participants  and  discussion 
respectively.  The  discussion  focus  must  remain  on  the  hypothesis  and  its  factor 
relationships  during  the  test.  After  the  test,  the  discussion  must  refocus  on  why  the 
hypothesis  is  untrue  or  cannot  be  proven  imtrue. 

The  constructive  simulation  must  be  initialized  with  the  same  parameters  as  used 
by  the  training  system.  The  simulation  situation  parameters  are  the  factors  that  are 
manipulated  by  the  trainees  to  discover  interrelationships.  These  are  ftxen  input  into  the 
simxilation  by  the  AAR  leader.  The  trainees  must  also  specify  the  measure  of 
performance/outcome,  the  dependent  variable,  that  will  dictate  whether  the  hypothesis  is 
to  be  accepted  or  rejected.  Hence,  the  testing  component  must  be  able  to  simulate 
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objective  actions  and  qnantifiable  results  of  a  battle  and  output  those  results  to  a  display 
system. 

The  testing  component  is  controlled  by  the  AAR  leader  and  has  the  following 
control  features:  play,  fast-forward/reverse  to  the  next  time  step,  fast-forward/reverse  to 
the  next  event,  and  pause.  Additionally,  die  component  should  allow  the  test  simulation 
to  be  started  at  any  point  in  the  simulation.  These  features  are  available  by  a  window 
menu  from  the  control  workstation. 

If  the  output  is  process  or  situation  based,  the  output  should  be  displayed  on  the 
Laser-based  3D  volumetric  display  system.  In  other  words,  the  3D  volumetric  display  is 
used  if  the  measured  outcome  is  dependent  upon  creating  specific  enemy-fiiendly  time- 
space  relationships.  This  will  focus  attention  on  the  results  of  the  test.  It  also  helps  the 
trainees  perceive  the  test  as  valid  because  the  actions  portrayed  in  3D  images  looks 
realistic  and  similar  to  the  replay  of  their  original  training  exercise.  If  statistical  in  nature, 
the  simulation  output  should  be  displayed  on  the  2D  presentation  medium. 

AAR  Leader 

As  in  all  other  AAR  systems,  the  AAR  leader  in  the  proposed  system  is 
responsible  for  complying  with  the  doctrinal  requirements  outlined  in  FM  25-100,  FM 
25-101,  and  TC  25-20.  (Chapter  1, 16  through  24)  However,  the  proposed  system 
provides  more  resources  for  the  AAR  leader  to  accomplish  this  guidance.  The  Inquiry 
tutor,  a  responsive  3D  display  system,  and  the  control  station  operator  combine  to  reduce 
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the  workload  of  the  AAR  leader.  Still,  the  AAR  leader  must  brief  the  purpose  and 
objectives  of  the  AAR  and  begin  the  discussion.  Once  started,  the  AAR  leader  can 
devote  himself  to  increasing  participation  in  the  discussion  and  allow  the  Inquiry  process 
to  focus  and  guide  the  discussion. 

The  mission  of  the  AAR  leader  is  to  prompt,  monitor,  and  reward  discussion 
participation  in  order  to  increase  the  effectiveness  of  the  AAR.  He  does  this  by 
establishing  participation  as  an  objective  of  the  AAR  discussion  in  the  introduction  and 
providing  participation  feedback  to  the  group  and  individual  trainees.  He  prompts  the  use 
of  the  discussion  mediums  to  engage  trainees  who  are  hesitant  and/or  cannot  articulate 
their  point  in  the  discussion.  At  the  same  time,  he  monitors  the  discussion  to  determine  if 
most  of  the  trainees  are  contributing.  If  someone  can  contribute  to  the  discussion  and  is 
not  doing  so,  he  solicits  their  comments  and  then  encourages  futme  participation  with 
laudatory  remarks. 

While  enhancing  participation,  the  AAR  leader  has  four  other  responsibilities  that 
also  require  him  to  monitor  the  discussion.  These  responsibilities  are  the  control  of;  the 
data  collection  subsystem,  the  Inquiry  tutor,  testing  component,  and  AAR  summary. 

The  first  is  to  control  and  input  information  queries  into  the  data  collection 
subsystem  to  support  the  AAR  discussion.  Initially,  the  AAR  leader  will  present  a  choice 
of  problem  areas  for  the  trainees.  These  potential  discussion  issues  are  derived  from  the 
data  associated  with  the  performance  standards  of  the  collective  training  tasks;  the  BDA 
or  results  from  the  exercise.  The  training  tasks  are  specified  in  the  unit’s  training 
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objectives.  This  provides  the  trainees  a  choice  of  discussion  starting  points.  Also,  the 
AAR  leader  presents  information  on  the  different  variables  and  factors  that  are  being 
discussed  in  an  effort  to  reveal  what  influenced  those  variables  and  factors  and  help 
trainees  to  establish  cause  and  effect  relationships. 

The  control  of  the  presentation  medium  is  an  implied  task  that  follows  from 
inputting  requests  to  the  collection  subsystem  and  trainee  inquiry  actions  to  the  tutor. 
Obviously,  the  AAR  leader  is  responsible  for  what  is  displayed  on  the  presentation 
medixim.  Therefore,  he  must  inteipret,  articulate,  and  elaborate  the  information  presented 
for  the  trainees.  The  data  collection  subsystem  output.  Inquiry  tutor  output,  and 
hypothesis  testing  results  are  the  three  basic  categories  of  information  presented  on  the 
presentation  medium. 

The  AAR  leader’s  second  responsibility  is  to  control  the  input  of  trainee  actions 
into  the  Inquiry  tutor.  Overall,  the  AAR  leader  implements  the  questioning  authority 
Inquiry  strategy  (Chapter  IV,  165)  to  make  participants  self-reliant  in  problem  solving 
discussion.  He  must  be  capable  of  articulating  and  elaborating  the  inquiry  process  by 
providing  the  appropriate  Inquiry  strategy  as  an  example  for  the  trainees.  The  AAR 
leader  can:  select  positive  and  negative  exemplars  and  vary  cases  systematically  to 
demonstrate  the  relationship  between  relevant  factors  and  show  how  factor  relationships 
are  constructed  into  a  rule.  This  responsibility  requires  a  working  knowledge  of  the 
Inquiry  approach. 
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The  AAR  leader’s  third  responsibility  is  to  input  the  variable  parameters  and  run 
the  testing  component  to  test  hypotheses.  The  AAR  leader  also  must  determine  if  the 
variable  parameter  adjustments  to  vary  cases  systematically  or  consider  alternative 
predictions  are  realistic.  This  determination  is  made  by  considering  the  hypothesis  itself 
and  the  purpose  of  the  test.  For  example,  assume  the  hypothesis  is  that  the  movement 
formation  affects  the  number  of  casualties  received  in  the  initial  contact  with  an  enemy. 
Then,  different  movement  formations  should  be  tested  against  the  same  enemy  formation, 
terrain,  vegetation,  and  visibility  conditions  as  given  in  the  original  training  exercise. 

The  movement  formation  is  the  factor  that  is  varied  systematically  while  other  variables 
and  factors  are  held  constant.  The  trainees  shordd  not  be  able  to  degrade  the  enemy  force 
nor  decrease  visibility  unless  die  purpose  of  the  test  changes.  An  appropriate  purpose 
might  be  to  discover  how  those  variables  interrelate  with  movement  formation  to  affect 
fiiendly  casualties. 

The  AAR  leader  must  also  guard  against  the  unrealistic  adjustment  of  the 
parameters  for  testing.  In  other  words,  groimd  vehicles  should  not  be  able  to  fly  or  be 
given  indestructible  armor  when  the  trainees  generate  hypothetical  cases  (Chapter  IV, 
160).  Additionally,  the  AAR  leader  must  ensure  that  the  trainees  are  allowed  to  alter  only 
the  variables  that  they  control.  He  will  have  to  determine  realistic  from  unrealistic 
adjustments  because  the  trainees  must  also  be  allowed  to  generalize  their  hypotheses. 
Generalizing  hypotheses  entails  testing  hypotheses  with  other  mission,  enemy,  terrain, 
troops,  and  time  conditions  (factors). 
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The  AAR  leader’s  final  responsibility  is  to  summarize  the  trainees’  hypotheses 
and  test  conclusions.  If  the  Inquiry  process  is  properly  implemented,  the  trainees  will 
have  identified  the  specific  causes  of  performance  outcomes  and  the  situational  training 
exercise  parameters  in  which  to  execute  re-training.  These  sets  of  task,  conditions,  and 
standards  form  the  imit’s  future  training  plan  and  are  the  important  results  of  an  effective 
AAR. 

Participants 

The  AAR  participants  are  the  unit  trainees  and  the  OPFOR  unit  leaders.  There 
can  be  as  many  as  36  and  as  few  as  8  trainees.  The  OPFOR  leaders  will  range  in  number 
from  1  to  6.  For  company  and  platoon  training  systems,  these  participants  will  range  in 
rank  and  experience  levels  from  private  to  captain  and  from  1  to  16  years  respectively. 

All  participants  must  be  seated/positioned  to  facilitate  discussion  amongst 
themselves.  Therefore,  die  participants  must  be  seated  facing  each  other  and,  at  the  same 
time,  have  easy  access  to  each  discussion  medium.  (See  Figure  E-2,  Physical  Layout) 

This  means  that  the  AAR  leader  is  not  the  center  of  attention;  everyone  is  not  facing  him 
throughout  the  AAR  and  he  is  not  behind  a  podium  nor  on  an  elevated  stage. 

The  OPFOR  provide  exercise  information  to  explain  what  happened  and  why. 
This  information  is  important  in  providing  insight  into  variables  and  their  relationships 
that  affected  performance.  They  do  the  same  for  OPFOR  actions  on  the  constructive 
simulation.  There  is  a  problem  if  the  OPFOR  are  live  trainees  also.  In  other  words,  if  the 
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training  exercise  consisted  of  two  platoons  in  networked  simulators  fighting  each  other 
on  a  virtual  battlefield,  two  separate  AARs,  one  for  each,  may  have  to  be  conducted.  In 
that  case,  leaders  from  each  platoon  could  attend  the  other  AAR  if  scheduled  at  different 
times. 

The  trainees  use  a  guided  inquiry  approach  to  improve  performance.  The  trainees 
also  select  the  problem  issue(s)  on  which  to  focus  the  AAR.  The  trainees  select  the 
problem  topic(s)  because  they  know  their  own  strengths  and  weaknesses  the  best.  They 
should  select  a  recurring  or  systemic  problem  to  discuss  rather  than  a  performance  failure 
that  occurred  by  chance.  These  are  normally  associated  with  the  performance  standards 
specified  by  the  training  objectives. 

AAR  Recorder 

The  AAR  recorder  component  consists  of  a  remotely  controlled  video  camera  and 
a  VCR  that  record  the  AAR  for  the  unit’s  take  home  package.  The  video  camera  is  wall 
mounted  and  responsible  for  recording  all  discussion  during  the  AAR.  It  also  captures 
the  sources  of  discussion  on  the  videotape.  The  VCR  records  the  information  displayed 
on  the  presentation  medium  to  videotape.  Of  special  interest  is  the  recording  of  the 
trainees’  hypothesis  formulation  and  testing  (of  solutions  developed.  Chapter  III). 
Captured  on  video,  the  unit  can  later  analyze  the  process  for  inquiry  errors,  factor 
relationship  misconceptions,  and  faulty  reasoning.  However,  most  important  of  all  are 
the  problem  solutions  in  the  form  of  future  training  tasks,  conditions,  and  standards 


278 


(solutions  planned.  Chapter  III)  that  have  been  verified  with  the  constructive  simulation 
of  the  testing  component. 
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